See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/220895964

Towards Profitable Virtual Machine Placement in the Data Center

Conference Paper · December 2011

DOI: 10.1109/UCC.2011.28 · Source: DBLP

CITATIONS READS
50 329

2 authors, including:

Weiming Shi
Georgia Institute of Technology
7 PUBLICATIONS   88 CITATIONS   

SEE PROFILE

All content following this page was uploaded by Weiming Shi on 28 May 2014.

The user has requested enhancement of the downloaded file.

 2011 Fourth IEEE International Conference on Utility and Cloud Computing

Towards Profitable Virtual Machine Placement in the Data Center

Weiming Shi and Bo Hong

School of Electrical and Computer Engineering
Georgia Institute of Technology, USA
Email: {weimingshi,bohong}@gatech.edu

Abstract—Motivated by the limit on the power usage ef-
fectiveness (PUE) of the data centers, the potential benefit
of the consolidation, and the impetus of achieving maximum
return on investment (ROI) on the cloud computing market,
we investigate VM placement in the data center, formulate
a multi-level generalized assignment problem (MGAP) for
maximizing the profit under the service level agreement and the
power budget constraint based on the model of a virtualized
data center, and solve it with a first-fit heuristic. Numerical
simulations show that the first-fit heuristic is effective in
solving the large-scale instances of the MGAP with the sampled
simulation setups.
Keywords-Profit Maximization; VM Placement; Data Center;
Multi-level Generalized Assignment Problem;
I. I NTRODUCTION
Large-scale data centers are constructed to meet the
ever-increasing demand for computing power from various
domains such as enterprise computing and cloud computing.
Typically, a data center is composed of power systems,
cooling systems and computing systems [1]. Maintaining
such large-scale infrastructures consumes enormous amount
of energy and imposes a large carbon footprint on the
environment. It was estimated that the power cost of a
data center would exceed the cost of the original capital
investment by 2012, and that the carbon footprint of data
centers would exceed the airline industry by 2020 [2]. With
these trends, it would be inadvisable to keep the data centers
underutilized, which would unfortunately happen to some
extent without careful planning in the data centers.
Meanwhile, the IT industry is undergoing a paradigm
shift towards leveraging the cloud computing infrastructure
to augment or support their business. In one of the service
models, namely IaaS (Infrastructure as a Service), computing
resources are virtualized and provisioned as services to
the customers who may further leverage the provisioned
infrastructure to provide domain-specific services to their
customers. From the perspective of the service providers,
the data centers face the problem on how to maximize
their profit given such large amount of energy costs for the
operation of the data center. The profit is equal to the revenue
minus the cost. The revenue is gained by provisioning the
services to the clients, while the cost is incurred for the
capital investment (CI) during the initial construction stage
and for the maintenance and the service provision during
the operation stage of the data center. Intuitively, the higher
performance levels the services are provisioned, the more
revenue is gained, but unfortunately the more energy costs
are incurred. Hence, it is necessary for the data centers to
operate under an advantageous operating configuration to
gain profit, wherein the revenue overtakes the cost.
It is well known that the revenue depends on the pricing
model of the service provider. However, how to build up
a competitive profitable pricing model depends on various
factors which are more relevant to the market economy
and the business model, and is out of the scope of our
paper. Given the fixed pricing model and the incurred CI,
a viable way for the data center to achieve the maximum
ROI is to reduce the operational cost by operating under
an efficient operating configuration, which clearly needs a
careful planning.
One way to increase the profit is to reduce the energy
costs, as is evidenced by the efforts such as improving
the power efficiency of the power systems in the data
centers; advocating the PUE (Power Usage Effectiveness)
initiated by Green Grid [3] as a metric towards building
energy-efficient, environment-friendly data centers; adopting
the virtualization-enabled solution (VMware [4], Xen [5],
KVM [6]) to increase the utilization of the servers in the
data center. These attempts reduce the energy costs for a
fixed operating configuration of the data center to certain
extent. However, the efficiency of these methods is gradually
approaching the theoretic limit. For example, the power
efficiency of the state-of-the-art UPS (Uninterruptible Power
Supply) system is approaching 95% and the transformers
and PDUs (Power Distribution Units) 98% [7]. The semi-
conductor chips, the horse power of the data centers are also
hitting the so-called brick wall by Moore’s Law wherein the
power consumption is expensive [8].
Therefore, there are strong economic incentives for the
decision makers of the data center to consider maximizing
the profit under the limited power budget. In this paper, we
try to approach this problem by exploring the dimension of
VM placement in the data center in the hope of maximizing
the profit without violating the service level agreement
(SLA) and the power budget. To that end, we first set up
a model of a virtualized data center, and then formulate
an optimization problem which aims to maximize the profit
under the constraints of the performance requirement of the

978-0-7695-4592-9 2011 138

U.S. Government Work Not Protected by U.S. Copyright
DOI 10.1109/UCC.2011.28
clients and the affordable power budget of the data center
infrastructure.
The rest of the paper is organized as follows. In Section II,
we provide the background information on our work. In
Section III, we describe the system model which lays down
the foundation for our discussion. In Section IV, we state the
problem and formulate a multi-level generalized assignment
problem (MGAP) to address the need of maximizing profit
subjective to the SLA and power budget constraint. We
provide one effective heuristic for the problem in Section V
and show the simulation results in Section VI. Section VII
summarizes the related work. Some concluding remarks are
given in Section VIII.
II. BACKGROUND I NFORMATION
Virtual machine is referred to as VM, and physical ma-
chine is referred to as PM throughout the paper. Here we
introduce the life cycle of a VM in IaaS model and the VM
placement and consolidation for the resource planning of the
data centers.
A. VM Life Cycle in IaaS Model
In IaaS model, the life cycle of a VM starts with
the request from the client stating the configuration, the
performance requirement and the life time. This request
then goes to the data center probably through the interface
of the web services. After receiving the request, the data
center management system starts searching the resource
pool, matching the available physical resources with the
requirement, and provisioning the requested VM to the
client. Then the client gets the control of the VM and starts
running the workloads on it. During the life time, the data
center management system will try to provision the VM
at the required performance level requested by the client.
When the requested life time expires, the VM is released
and the physical resources associated with the VM is freed
and becomes available for allocation to other VMs again.
B. VM Placement and Consolidation
Consolidation is a potential way for the data centers to
improve the system utilization, reduce the power footprint
and increase the profit. It can be conducted at the start of the
life time of the VMs when the management system needs
to decide how to map the physical resources of the PMs to
the provisioned VMs or at a regular frequency after some
PMs have already been allocated to provision the VMs. The
former is referred to as the VM placement and the latter
as the VM consolidation in the literature. In both cases,
a set of candidate VMs are needed to be assigned to a
set of candidate PMs to fulfill the performance requirement
subject to the resource and SLA constraints. The difference
is the initial configuration of the system when the decision
is made. VM consolidation needs to take into account the
migration cost of the VMs while VM placement needs not.
139
After consolidating multiple VMs onto a small number of
PMs, the extra idle PMs that would cost a large fraction
of operational energy costs can be shut down. Although
consolidation helps to save the energy costs, the degraded
system performance would lead to the revenue losses.
It is natural to relate the VM placement problem to the bin
packing problem (BPP), if the resource consumption of the
VM is treated as the item size and the resource capacity of
the PM as the bin size. The object is to pack as many VMs as
possible onto a small number of PMs so as to minimize the
number of operating PMs. But certain subtleties differentiate
it from BPP: 1) the resource consumption could be multi-
dimensional in terms of the CPU time, the memory, the I/O
and the power and may not be simply additive; 2) the SLA
of VMs makes the sizes of VMs to be elastic; 3) packing
as many as VMs onto a single PM may not be advisable in
terms of the power budget and QoS as noted in the works
[9] [10] where certain power efficient operating points exist
for each PM.
Moreover, other challenges exist: 1) VMs and PMs can be
heterogeneous. Each has different performance requirement
and would incur different resource budget on the host PMs.
The incurred resource budget not only depends on the
performance requirements of guest VMs, but also depends
on the characteristics of the host PMs. 2) The result of the
consolidation differs from different optimization objectives.
For example, if the interconnect latency between two VMs
is to be minimized, they can be consolidated onto a single
PM. However, if the ensemble power consumption of those
two VMs is to be minimized, it is possible to assign them
to different PMs due to their different power characteristics.
In this work, we try to address these issues. Instead of
minimizing the power consumption, we treat the power
budget as a special kind of system resource, quantify the
elasticity of the VMs by the predefined performance levels,
and endeavor to maximize the profit that can be obtained
by provisioning the VMs to the clients without violating the
SLA.
III. S YSTEM M ODELS
A. A Model of a Virtualized Data Center
The model of the virtualized data center under our con-
sideration is composed of a set of interconnected PMs. For
the simplicity of the discussion, we assume that the data
center has a single-level hierarchy and that there are two
levels of system managers, i.e., the global and the local
manager, which are responsible for managing the resources
in the data center. When the VM requests from clients
come, they are first buffered in a service queue before being
serviced. The global manager conducts the admission control
and deploys the VM requests to the selected PMs based
on the current management purpose of the data center and
the system information collected from the PMs. The local
manager is responsible for implementing the decision of
the global manager by scheduling the set of VMs on the
assigned PM. It also accounts for monitoring and collecting
the information needed by the global manager such as the
aggregated resource capacity and the power consumption.
B. Service Level Agreement (SLA)
1) SLA Model: The service level agreement (SLA) is
a part of the contract between the clients and service
providers. It formally defines the levels of the services and
the potential profits the service providers may obtain if the
service is delivered to the clients at the agreed levels. From
the perspective of the data center, the SLA serves as the
constraint for its optimization purpose. The performance
metrics in the SLA could be defined for an ensemble of
clients or per client. In the context of cloud computing,
various performance metrics of SLAs exist such as the
averaged response time, the throughput, the mean time
between failures (MTBF), etc.
For the best interests of the clients, we consider a per-
formance metric defined per client in our paper. For IaaS
model which obtains the profit by provisioning the VMs,
we take the following SLA model wherein the performance
requirement of clients is abstracted as certain predefined
performance levels K = {0, 1, · · · , K} of the VMs, and
the associated revenue of each provisioned VM is related to
its runtime performance level k ∈ K provisioned by the data
center.
Service Level Agreement (K=4)
100
linear model
80
sublinear model
superlinear model
Revenue
60
40
20
0
0 1 2 3
Performance Level (k)
Figure 1. SLA Model
Figure 1 illustrates three different pricing models (K = 4)
as a part of the SLA model. The revenue suffers certain loss
if the provisioned performance level is below the required
level but is still acceptable. Level k = 0 is reserved for the
out-of-service which obtains no revenue. The revenue of the
illustrated pricing model is a linear, sublinear and superlinear
function of the performance level k ∈ K respectively, which
we believe are versatile enough to fit a wide range of pricing
models. The actual function forms of the pricing models may
be tuned based on the need of the decision maker and the
business purpose.
2) SLA Violation Metric: The elasticity of the VMs plays
an important role in the VM placement problem. The more
elastic the VMs are, the higher the chance that more VMs
can be consolidated onto one PM. Hence we consider two
kinds of the SLAs: the hard SLA and the soft SLA. For the
hard SLA, if the VMs are not provisioned at the requested
levels, then no revenue will be obtained. For the soft SLA,
clients are tolerant to the degraded performance levels to
some extent, but the obtained revenue suffers certain loss
according to the pricing model until no revenue will be
obtained if the provision performance level is beyond the
tolerance of the clients, which corresponding to k = 0.
For instance, if the service levels of the initiated VMs are
set at k = 3, then the performance levels can only take
two values, i.e., k = {0, 3} for the hard SLA while the
performance levels can be k ∈ {0, 1, 2, 3} for the soft SLA.
These two kinds of SLAs together with their mixed variants
(with different hard and soft SLA ratios) are versatile enough
to model a wide range of runtime scenarios of the data
centers.
We define the SLA violation ratio as follows:
SV R = 1 − |J  |/|J| (1)
where J  = {j : k(j) = k, j ∈ J}. |J  |/|J| is the hard
SLA rate, and SV R represents the percentage of the VMs
that are provisioned at the performance levels lower than the
requested levels.
C. System Resource
VM placement inherently involves multiple system re-
sources. Here we identify the power budget of a PM as
an important single resource whose consumption deserves
careful planning with the power capping mechanism in the
data centers. The power usage per VM is related to various
system resources such as CPU, memory, I/O, etc. The study
on the VM power usage indicates that the power usage is
mostly related to load level of the VM, i.e., the utilization
of CPU and the memory [11] [12]. So we calculate the
power consumption of the host PM with the following power
4 model:
P = a + e · u/U (2)
where the first term a is the idle power and the second term
e · u/U is the dynamic power, which is further determined
by the VM load level u/U of the host PM. Here idle
power a and the maximum dynamic power e are predefined
constants determined by the power efficiency of the PM; u
is the amount of consumed computing resource, and U is the
computing resource capacity, full utilization of which will
consume the maximum dynamic power e.
For example, if only the CPU resource is considered, and
suppose that there are c processor cores C = {1, 2, · · · , c}
on one PM, each operating at the frequency fc with an IPC
(instructions per clock) value IP Cc , and the PM runs for T
time units, then the computing resource of the PM can be
defined as: 
U= IP Cc · fc · T (3)
c∈C
140
U is essentially the number of instructions that can be – the dynamic power corresponding to 100% load
committed by the PM and hence consumed by the hosted level ei = (1 − αi )Pi
VMs in T time units. – the power consumption pi
A constraint on the power budget can implicitly con- – the computing resource capacity Ui
strain the consumption of the system resources such as the – the operational power cost rate mi
computing resource usage in (2). Hence, we can treat the – the power budget bi , bi ≤ Pi
power budget as a kind of aggregated resource that implicitly • each VM j ∈ J has the following attributes:
incorporates other system resources in the power model. – the performance level k ∈ Kj
Note that our model could also apply to the general sce- – the computing resource usage uijk
nario wherein a detailed additive resource model is available – the revenue rijk
to map the utilized system resources to a single metric
We assume that the operational cost of the data center is
that quantifies the resource consumption u and capacity
mainly attributed to the power cost of the PMs. The problem
U . However, some efforts are required to minimize the
is to maximize the total profit with a feasible assignment of
information loss incurred in the mapping. For instance,
J to I. An assignment is essentially a mapping from J to I
vastly different amount of resource utilization of different
which can be described by the decision vector x = (xijk ),
system resources (e.g., the memory and the CPU) could
i∈I,j∈J,k∈Kj .
generate the same amount of power consumption because of 
the different power efficiency of the system resources. Hence 1 if PM i provisions VM j at level k
xijk =
the single metric (e.g. the power usage) alone impedes us 0 otherwise (4)
from identifying the exact resource utilization of the VM
Hence, the VM placement problem can be formulated as
(e.g. whether the power usage is mainly attributed to the
a mixed integer programming (MIP) problem:
memory or the CPU). This kind of information loss could be   
alleviated if we assume the available system configurations Maximize R = rijk xijk − mi pi (5)
(e.g, the number of CPU cores and the amount of memory in i∈I j∈J k∈Kj i∈I
the above case) of the VMs are known beforehand and that such that p i ≤ bi ∀i∈I (6)
a proper weighting function can be designed to account for ei  
the difference in the power efficiency of the different system pi = a i + ( uijk xijk ) ∀i∈I (7)
Ui
resources before mapping the multiple resource utilization to j∈J k∈Kj

a single metric. xijk = 1 ∀j∈J (8)
IV. P ROBLEM F ORMULATION i∈I k∈Kj

Here we consider the VM placement problem with the xijk ∈ {0, 1} ∀i∈I,∀j∈J,∀k∈Kj (9)
objective of maximizing the total profit subject to the SLA Equation (5) defines the profit as the revenue minus the
and the separable power budget constraint. VM placement power cost. Constraint (6) ensures that each PM operates
is conducted by the global manager at the level of the data under its power budget. Equation (7) is the power model
center. we stated in Section III-C. Constraint (8) ensures that each
We focus on the decision making at the data center VM is assigned to only one PM and operates at only
level and assume that the local manager can fulfill the one performance level. Constraint (9) indicates the decision
inclination of the global manager by scheduling the assigned variable xijk is binary. All the parameters are implicitly
VMs properly under the separable power budget assigned assumed to take non-negative rational numbers.
to each PM. For the simplicity of our discussion, we also To preclude trivial solutions, we further assume that:
assume that we can omit the overhead caused by the system
ei
managers. ai + uijk ≤ bi ≤ Pi ∀i∈I (10)
Ui
The mathematical notations used in the paper are intro- ei 
duced as follows: ai + min uijk > bi ∀i∈I (11)
Ui k∈Kj
• the set of candidate PMs: I = {1, 2, · · · , I} j∈J

• the set of candidate VMs: J = {1, 2, · · · , J}
• the set of performance levels: Kj = {0, Kj } int the
hard SLA and Kj = {0, 1, 2, · · · , Kj } in the soft SLA
• each PM i ∈ I has the following attributes:
– the peak power Pi
– the idle power rate αi , the lower the value, the
higher the power efficiency
– the idle power ai = αi Pi
Constraint (10) ensures that the resource of one PM is
enough to provision one VM at any performance level.
Constraint (11) precludes the situation wherein all the VMs
can be consolidated onto one PM.
V. S OLUTION M ETHOD
The formulated problem can be reduced to a multi-level
generalized assignment problem (MGAP), if the VMs are

141
treated as the tasks, the PMs as the agents, and the power Algorithm 1 First-fit Heuristic
budget as the resource. MGAP is a variant of the well- Ju = J : the set of unallocated VMs
studied generalized assignment problem (GAP). It considers Ip = ∅ : the set of candidate PMs
the problem of assigning the set of tasks of the multiple in = 1 : the id of the next PM to be powered-on
performance levels to the set of agents so that each task is pijk : the dynamic power of VM j of level k on PM i
assigned to only one agent at one performance level and fijk : the profit of VM j of level k on PM i
the resource capacity of each agent is not violated. It is NP- bi = bi : the residual power budget
hard and even the problem of determining whether a feasible pm : the minimum power usage of VM
solution exists is NP-complete [13].
In our problem, we assume the computing resource usage Require: xijk = 0
of the VMs takes deterministic values. But our model is 1: while Ju = ∅ do
general enough to incorporate the stochastic nature of the 2: j = Ju .next();
underlying problem where the actual distribution of the com- 3: ij = the first PM ∈ Ip ∪ in satisfying the SLA
puting resource usage of the VMs uijk could be estimated 4: if ij is not powered on then
through data collection and statistical inference in the data 5: Power on PM ij and add it to Ip
center. In the stochastic case, we can derive a deterministic 6: bij = bij − ipij
equivalent (DE) problem based on the proportional mean 7: in = in + 1
variance model proposed in [14] by assuming that uijk is 8: end if
normally distributed with known mean and variance, i.e., 9: xij jkj = 1
2
uijk ∼ N (ūijk , σijk ), which depicts the scenario that no 10: bij = bij − pij jkj
users are under-utilize or over-utilize the allocated VMs 11: update the profit with fijk
given the mean value as the required performance level. 12: if bij < pm then
For small-scale instances of MGAP, exact solutions are 13: delete ij from Ip
accessible by leveraging the MIP solver that implements 14: end if
branch-and-bound methods such as provided by CPLEX 15: delete j from Ju
[15]. But such existing exact solution techniques do not scale 16: end while
to the large-scale instance that models the data center. There- 17: calculate the number of powered-on PMs, the profit
fore, we design a problem-specific heuristic algorithm. There 18: calculate the SLA violation rate
exist some heuristic algorithms such as [16] [17]. We notice
that the performance of the first-fit heuristic matches the
more complicated heuristics in [16] in the sampled system second set is to investigate the sensitivity of the solution to
setups. The complexity of the first-fit heuristic is lower than the power efficiency, the SLA violation ratio, and the pricing
those heuristics, and the approximation ratio is within 20% model.
of the optimum solution based on our simulation results. For the simplicity of the simulation, all the PMs are set to
Hence, we explore the properties of the VM placement in be homogeneous in terms of the power efficiency. Our model
the data center based on the result of the first-fit heuristic. is general enough to consider heterogeneous PMs where the
The heuristic is listed in Algorithm 1. It iterates the list extra step needed is an normalization of the power usage
of the unallocated VMs and tries to assign the VMs to PMs of the VMs versus the power efficiency related parameters
which have extra power budgets to provision them. Line 3 of the heterogeneous PMs. Hence, the peak power Pi is set
finds the first-fit PM that satisfying the power budget and to be 650W and the idle power rate αi is sampled in the
the hard/soft SLA constraint from the list of the candidate range [0.3, 0.7] for all PMs. The power cost rate mi is also
PMs. Line 4-8 decide whether the chosen PM is powered set to take the same value $0.01/kWh, although it can take
on or not, if not, power it on. Line 9-11 update the decision different values to model the situation of the multi-electricity
variable and the state variables. Line 12-14 remove the market when the data center is distributed in different areas.
specified PM from the list of the candidate PMs as it can The function forms we used for the linear, sublinear and
not even provision any VM at the lowest performance level. superlinear pricing models are r = 10k, r = 10ln(1 + k),
Line 17-18 calculate the results of the performance metrics r = 10(1 + k)ln(1 + k) respectively.
VI. S IMULATION R ESULTS We define one unit of computing resource be equal
A. Simulation Setup to the resource consumption of one VM provisioned at
performance level k = 1, and one unit of power budget be
We conduct two sets of simulations. The first set aims to
equal to the dynamic power usage of one VM provisioned at
estimate the approximation ratio of the first-fit heuristic and
performance level k = 1 when the idle power rate αi = 0.3.
verify that it is suitable for solving the formulated MGAP
The computing resource capacity of each PM U is set to
which models the VM placement in the data center. The
be 16 units of computing resource, and the power budget of

142
each PM bi to be 16 units of power budget. where Mr and Nr are the realization of the profit and the
The number of the available performance levels K are number of the powered-on PMs in the simulation. Note that
set to 4. The revenue is designed to be proportional to AR(FF) is over-estimated in this way as the VMs are not
the performance level k of the VM. The workload, i.e., divisible in reality.
the computing resource usage of each VM is assumed to To have Mub /Mr and Nr /Nlb under various system
have a normal distribution with the mean value linearly setups, simulations are run with the idle power rate α
proportional to its performance level k. Given the number sampled in the range [0.3, 0.7] with the granularity of 0.1,
of VMs J, Jk the distribution of the number of instances of the hard SLA ratio set to 0, the number of VMs J sampled
each level k ∈ {1, · · · , K} is set to be inversely proportional at 2i , i = [7, 15], i ∈ Z and the superlinear pricing model
to k, i.e., Jk = (1 − 2k/(K(K + 1)))J to model the non- (corresponding to the largest profit loss).
increasing change of the demand with the increase of the The results of Mub /Mr and Nr /Nlb are shown in the left
price (proportional to k). subplot and the right subplot of Figure 2 respectively. Each
Each data point is the average result of 10 runs of the curve corresponds to a sampled α. Note that we use the
simulation. superlinear pricing model to estimate Mub /Mr , but better
results are present for sublinear and linear pricing models,
B. Complexity and Quality of the Solution and are omitted in the left subplot for brevity.
AR(FF) takes the maximum of these two values under the
1.06 1.2
1.19
same system setup. It can be seen that the first-fit heuristic
the Number of Powered−on Machines

1.05 1.18 approaches the optimal values of the large-scale instances of

Approximation Ratio i.t.f the Profit

α=0.3 α=0.4 α=0.5

the MGAP (within 20%), and that a lower value of α leads to
1.17
α=0.6 α=0.7
Approximation Ratio i.t.f

1.16
1.04
1.15
1.14
lower AR(FF) because lower α provides more power budget
1.03
1.13 and has the higher chance to consolidate multiple VMs onto
1.12
1.02
1.11 one PM.
1.1
1.01 α=0.3 α=0.4 α=0.5
α=0.6 α=0.7
1.09
1.08
C. Sensitivity to the Power Efficiency and the Hard SLA
1
2^7 2^8 2^9 2^10 2^11 2^12 2^13 2^14 2^15
1.07
2^7 2^8 2^9 2^10 2^11 2^12 2^13 2^14 2^15 Ratio and the Pricing Model
Number of VMs Number of VMs

1 1
Figure 2. Estimated Approximation Ratio
Normalized Number of Powered−on PMs

0.98
The complexity of the first-fit (FF) heuristic is O(IJ) in 0.98

of Superlinear Pricing Model

0.96

the worst case. With the proper setting of the threshold pm to 0.96
0.94

Normalized Profit
remove a PM from the candidate list Ip , the complexity can 0.94 0.92

be further reduced to O(J). FF is a simple online algorithm 0.9

0.92
for solving the bin packing problem with an approximation 0.88

ratio of 17/10 [18]. 0.9

α=0.3 α=0.4 α=0.5 0.86
α=0.3 α=0.4 α=0.5
α=0.6 α=0.7 α=0.6 α=0.7
To verify the quality of the solution by the first-fit heuristic 0.88
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0.84
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
when the VMs are elastic, we establish the upper bound Hard SLA ratio (HSR) Hard SLA ratio (HSR)

of the maximum profit Mub and the lower bound of the Figure 3. Sensitivity to the Hard SLA ratio, Idle Power Ratio
minimum required number of PMs needed to fulfill the
service Nlb , and try to estimate the approximation ratio of To test the sensitivity of the obtained solution to the power
the heuristic AR(FF) based on them. efficiency and the hard SLA ratio (HSR) and the pricing
Nlb is obtained by assuming the VMs are divisible across model, simulations are run with J = 215 , α sampled in
the PMs. the range [0.3, 0.7] with the granularity of 0.1, the HSR
ei  sampled in the range [0, 1] with the granularity of 0.1, and
Nlb = min uijk /U (12) the linear (resp. sublinear, superlinear) pricing models taking
i∈I (bi − ai )
i∈I j∈J k∈K the function forms described in VI-A. The simulation results
Mub is obtained by deducting the power cost of the are presented in Figure 3.
minimum number of powered-on PMs from the maximum The left subplot of Figure 3 shows the result of the
revenue for provisioning all the VMs. normalized number of powered-on PMs versus the hard
 SLA ratio of the VMs, normalized w.r.t the maximum value
Mub = max rijk − Nlb · min(mi · P Pi ) (13) obtained at HSR=1. The results of the normalized average
i∈I,k∈Kj i∈I
j∈J number of powered-on PMs are independent of the pricing
AR(FF) is estimated by models. Each curve of it corresponds to a different α value.
The right subplot of Figure 3 shows the results of the
AR(F F ) = max{Mub /Mr , Nr /Nlb } (14) normalized profit of the superlinear pricing model versus

143
the idle power rate, normalized w.r.t the maximum value with the same simulation setup in VI-C. Each bar in the
obtained at HSR=1. The first-fit heuristic manages to control plot corresponds to a sampled system setup with the bars
the profit loss within 15% in the worst case, and the profit corresponding to the same hard SLA ratio grouped together.
loss is further reduced to 5% for power efficient system For a particular HSR, the SLA violation rate is expected to
(α = 0.3). Basically, the higher the HSR, the less profit loss be 1-HSR. However, the first-fit heuristic manages to keep
is suffered. Better results and the same trend are present for the SLA violation rate under 15% in the worst case under all
the sublinear and linear pricing models but are omitted for sampled system setups, even the model has a large tolerance
brevity. to the SLA violation when the HSR is lower,
VII. R ELATED W ORK
Normalized Average Number of Powered−on PMs
1
1
0.95
0.99
0.98 0.9 Theoretical results related to this work can be found in
Normalized Average Profit

0.97
0.96
0.95
0.85
0.8
[18]–[20]. Generalized assignment problem (GAP) is known
0.94
0.93
0.75
0.7
to be APX-hard and has been shown to be e/(e−1)+ hard
to approximate for some constant  > 0 [19]. The first-fit
0.92
0.65
0.91
0.6

decreasing (FFD) heuristic has an approximation ratio of 11

0.9 Linear Pricing Model
0.89 Sublinear Pricing Model 0.55
0.88 Superlinear Pricing Model 0.5 9
0.87
0.86 0.45 which is shown to be tight [20]. The first-fit (FF) heuristic
0.85 0.4
0.3 0.4 0.5
Idle Power Rate α
0.6 0.7 0.3 0.4 0.5
Idle Power Rate α
0.6 0.7
does not require sorting as is required by FFD heuristic and
has an approximation ratio of 17 10 [18]
Figure 4. Sensitivity to Idle Power Ratio and Pricing Model
FFD heuristic has shown to be effective in [21], [22] for
Figure 4 shows the results of normalized average profit of mapping the physical resources to the virtual machines. Re-
the three pricing models and the normalized average number cently, Beloglazov et. al. compare several proposed heuris-
of PMs versus the idle power rate. Each data point takes tics [23] for the efficient energy management of the data
the average value of all the sampled data points of the center and present several open research challenges.
same idle power rate in Figure 3 and normalized w.r.t the While most of the research literature consider the problem
maximum value obtained at α = 0.3. The left subplot shows of power management in the virtualized data center, few
the three normalized average profit corresponding to the work targets at maximizing the profit from the perspective of
linear, sublinear and superlinear pricing models. The right the data centers. Two recent work with similar optimization
subplot shows the normalized number of powered-on PMs, objectives as ours are [24], [25]. In [24], the authors consider
which is independent of the pricing models. As we can see, a multi-dimensional SLA-based resource allocation problem
the solution of the sublinear pricing model is more sensitive based on the queuing model and solve it by a greedy algo-
to the idle power rate than the linear and superlinear pricing rithm. In [25], the authors also consider the VM placement
models, and the higher the power efficiency (i.e., the lower in data centers. But they consider an anti-collocation SLA
the α) of PMs, the less powered-on PMs are needed and the constraint, formulate the problem as a combinatorial auction
more profit the system will obtain. problem, and solve a relaxed linear programming problem
An extrapolation of this trend to α = 0 would seems to be with the column generation method. In our work, the pricing
counter-intuitive as the data center would be ‘free’ to power model takes discrete values that are proportional to the
on more servers than the case with a larger α. However, this required performance levels of the VMs, and hence be more
situation would not happen with the FF heuristic as it breaks realistic than the continuous pricing model based on the
the tie by not powering on new PMs if the residual resource actual resource utilization considered in [25].
is enough.
VIII. C ONCLUSION AND D ISCUSSION
D. SLA Violation Rate This paper studies the profit maximization problem of the
data center. We formulate an optimization problem from the
15%
14%
13%
α=0.3
α=0.4
perspective of the decision maker of the data center. Our
12%
SLA Violation Ratio

11% α=0.5
α=0.6
work not only presents a general model for the resource
10%
9%
8%
α=0.7 planning in the data centers, but also identifies first-fit as a
7%
6% simple yet effective heuristic for VM placement in the data
5%
4%
3%
center which promises to retain profit for data centers.
2%
1%
0%
Currently, we adopt the linear power model wherein the
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Hard SLA Ratio power of the hosted VMs is additive. In the future work,
a more detailed nonlinear power model wherein the power
Figure 5. SLA Violation Ratio consumption of the VMs would depend on the operational
SLA violation rates are independent of the pricing models. points of the PMs can be investigated. We also assume that
The results are also obtained and presented in Figure 5 the local manager can successfully schedule the assigned

144
 VMs to meet the SLA and power budget constraint. This
may not always be true. More accurate information could be
gained by leveraging the event-driven simulation framework
such as Cloudsim [26]. Besides, we formulate the problem
of a single scheduling period. With proper modeling of
the migration cost, our proposed model can be extended
to investigate the VM consolidation problem over a multi-
period horizon. Note that the modeling of the migration cost
may depend on the memory footprints of the VMs.
ACKNOWLEDGMENT
This work is supported by the US National Science
Foundation under award number CNS-0845583.
R EFERENCES
[1] L. Barroso and U. Hölzle, “The datacenter as a computer:
An introduction to the design of warehouse-scale machines,”
Synthesis Lectures on Computer Architecture, vol. 4, no. 1,
pp. 1–108, 2009.
[2] “Quick start guide to increase data center energy efficiency,”
GSA Public Buildings Service, Applied Research Program
and DOE Federal Energy Management Program, October
2008.
[3] C. Belady, A. Rawson, D. Pfleuger, and S. Cader, “Green
grid data center power efficiency metrics: Pue and dcie,”
Consortium Green Grid, 2008.
[4] C. Waldspurger, “Memory resource management in VMware
ESX server,” ACM SIGOPS Operating Systems Review,
vol. 36, no. SI, pp. 181–194, 2002.
[5] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho,
R. Neugebauer, I. Pratt, and A. Warfield, “Xen and the art
of virtualization,” in Proceedings of the nineteenth ACM
symposium on Operating systems principles. ACM, 2003,
pp. 164–177.
[6] A. Kivity, Y. Kamay, D. Laor, U. Lublin, and A. Liguori,
“kvm: the Linux virtual machine monitor,” in Proceedings of
the Linux Symposium, vol. 1, 2007, pp. 225–230.
[7] “Energy efficiency and environmental considerations for the
data center,” (White Paper) Anixter International, August
2009.
[8] K. Asanovic, R. Bodik, J. Demmel, T. Keaveny, K. Keutzer,
J. Kubiatowicz, N. Morgan, D. Patterson, K. Sen,
J. Wawrzynek et al., “A view of the parallel computing
landscape,” Communications of the ACM, vol. 52, no. 10,
pp. 56–67, 2009.
[9] S. Srikantaiah, A. Kansal, and F. Zhao, “Energy aware
consolidation for cloud computing,” in Proceedings of the
2008 conference on Power aware computing and systems.
USENIX Association, 2008, pp. 10–10.
[10] H. Van, F. Tran, and J. Menaud, “Performance and power
management for cloud infrastructures,” in 2010 IEEE 3rd
International Conference on Cloud Computing. IEEE, 2010,
pp. 329–336.
145
View publication stats
[11] B. Krishnan, H. Amur, A. Gavrilovska, and K. Schwan,
“VM Power Metering: Feasibility and Challenges,” ACM
SIGMETRICS Performance Evaluation Review, vol. 38, no. 3,
pp. 56–60, 2011.
[12] M. Pedram and I. Hwang, “Power and Performance Modeling
in a Virtualized Server System,” Power, 2011.
[13] F. Glover, J. Hultz, and D. Klingman, “Improved computer-
based planning techniques. part ii,” INTERFACES, vol. 9,
no. 4, pp. 12–20, 1979.
[14] D. Spoerl and R. Wood, “A stochastic generalized assignment
problem,” in INFORMS Annual Meeting, Atlanta, GA, vol. 19,
2003, p. 22.
[15] “Ibm ilog cplex optimizer,” (ac-
cessed July 2011). [Online]. Available:
www.ibm.com/software/integration/optimization/cplex-
optimizer
[16] A. French and J. Wilson, “Heuristic solution methods for
the multilevel generalized assignment problem,” Journal of
Heuristics, vol. 8, no. 2, pp. 143–153, 2002.
[17] A. Ceselli and G. Righini, “A branch-and-price algorithm
for the multilevel generalized assignment problem,” OP-
ERATIONS RESEARCH-BALTIMORE THEN LINTHICUM-,
vol. 54, no. 6, p. 1172, 2006.
[18] D. Johnson, A. Demers, J. Ullman, M. Garey, and R. Graham,
“Worst-case performance bounds for simple one-dimensional
packing algorithms,” SIAM Journal on Computing, vol. 3, p.
299, 1974.
[19] U. Feige and J. Vondrak, “Approximation algorithms for
allocation problems: Improving the factor of 1-1/e,” in Foun-
dations of Computer Science, 2006. FOCS’06. 47th Annual
IEEE Symposium on. IEEE, 2006, pp. 667–676.
[20] G. Dósa, “The tight bound of first fit decreasing bin-packing
algorithm is ffd (i) 11/9 opt (i)+ 6/9,” Combinatorics, Al-
gorithms, Probabilistic and Experimental Methodologies, pp.
1–11, 2007.
[21] N. Bobroff, A. Kochut, and K. Beaty, “Dynamic placement of
virtual machines for managing sla violations,” in Integrated
Network Management, 2007. IM’07. 10th IFIP/IEEE Interna-
tional Symposium on. IEEE, 2007, pp. 119–128.
[22] T. Wood, P. Shenoy, A. Venkataramani, and M. Yousif,
“Black-box and gray-box strategies for virtual machine mi-
gration,” in Proc. NSDI, 2007.
[23] A. Beloglazov, J. Abawajy, and R. Buyya, “Energy-aware
resource allocation heuristics for efficient management of data
centers for cloud computing,” Future Generation Computer
Systems, vol. In Press, Corrected Proof, 2011.
[24] H. Goudarzi and M. Pedram, “Maximizing profit in cloud
computing system via resource allocation,” in Intl workshop
on Data Center Performance.
[25] D. Breitgand and A. Epstein, “Sla-aware placement of multi-
virtual machine elastic services in compute clouds,” in 12th
IFIP/IEEE International Symposium on Integrated Network
Management (IM11), Dublin, Ireland, 2011.
[26] R. Calheiros, R. Ranjan, C. De Rose, and R. Buyya,
“Cloudsim: a novel framework for modeling and simulation
of cloud computing infrastructures and services,” 2009.