Virtual Machine Power Metering and Its Applications
Chongya Ma, Zhiying Jiang, Ke Zhang and Guangfei Zhang
Institute of Computing Technology,
Chinese Academy of Sciences,
Beijing, 100190, China
{ machongya, jiangzhiying, zhangke, zhangguangfei}@ict.ac.cn
Abstract—With the rapid growth of power consumption and
the cooling cost in data centers, it is more and more important to
improve the utilization rate of power by more sophisticate power
monitoring and management strategy. VM power consumption
measurement and related scheduling strategy contribute to make
cloud computing center more efficient on the basis of making full
use of existing power and cooling resources. In this paper, we
propose methods to meter the power consumption of VM, and
design scheduling strategies based on power consumption to
migrate VM between hosts in order to achieve power capping.
Specifically, we propose an improved VM power metering
method, in which the power consumption of cooling fans is
isolated. Besides, we solve the problem of the dynamic power
metering where the hyper-threading of host processor is enabled.
In addition, we propose VM migration and scheduling strategy
on the basis of power consumption. Last but not the least, we
propose a strategy to monitor data center status, including power
consumption and resource utilization of all hosts and VMs. All
above are demonstrated an evaluated in a prototype system.
Keywords—virtual machine; power metering; power capping
I. INTRODUCTION
With the rapid development of computer science and
technology, virtualization technology is more and more widely
used in data centers. Virtualization is also used in the platforms
of cloud computing. In the programming and test environments,
virtualization technology has many advantages, such as having
a lower total cost, improving availability, high-efficiency, and
so on; But at the same time it also brings great challenges. The
growing cost of power consumption and power cooling is a
huge challenge which both cloud computing and data center
operators are faced with. So providing a kind of effective VM
power consumption measurement strategy is imminent. Many
international famous IT companies have introduced advanced
technology to reduce the power consumption of the data center.
For example, Intel data center management platform (Data
Center Manager, DCM) can monitor and manage the energy
consumption of data center frame and the sever group.
In a data center, administrators use power capping to ensure
power supply. Power capping provides configuration
information of power limit for every sever so as to avoid the
excessive supply of energy and make the data center power and
cooling resources under effective control. Besides. the ability
to monitor the real-time status of all hosts and VMs, such as
CPU and memory utilization is of great importance.
Zhixiong Jiang, Chunyang Lu and Yushan Cai
Information Technology Center,
China National Petroleum Corporation,
Beijing, 102200, China
{jiangzhixiong, luchunyang, caiyushan}@cnpc.com.cn
Academics have always focused on studying how to
measure VM power consumption in the traditional server
environment and also put forward a variety of models and
methods. Finer-grained VM power consumption measurement
strategy, VM scheduling method and state monitoring method
can help achieve highly efficient and green cloud computing
center by using existing resources of VM power consumption
more accurately.
There are a lot of challenges in VM power consumption
measurement, VM scheduling, and state monitoring in cloud
computing center. The problem of VM power consumption
measurement is how to make the method more accurate.
Besides, in some special cases (hyper-threading function being
opened), the existing measuring methods (including the method
we have proposed before) is no longer valid, then how should
we deal with it. The challenges in VM scheduling are about
how to detect the host whose power consumption is close to the
limit, how to choose the VM that should be migrated and how
to choose the destination host where the VM should be
migrated.
The goal of our research is to carry out an improved VM
machine power metering method on the basis of VPower(a VM
machine power metering method we have proposed before).
Specifically, we isolate the power of fan and analysis it
independently. And we also analysis the VM power
consumption when hyper-threading function of the processor is
opened. According to data of VM power consumption, we
study the VM scheduling strategy. To be specific, through
monitoring the power consumption of the host and VM in real
time, we migrate the VM on the host whose power is close to
the limit to the host whose power is low, so as to reduce power
consumption in the source host and realize VM scheduling and
power capping. Finally, we also monitor the VM state, such as
CPU utilization, memory utilization, cache missing rate and so
on.
II. RELATED WORK
In our previous work[1], a method named VPower was
proposed. The energy consumption of a computer is divided
into two parts: static power and dynamic power. The static
power is obtained when the host OS is in idle state. To get the
dynamic power, a two dimensional lookup table(LUT) is
constructed for each VM, which gives an empirical power
value at certain CPU utilization and last-level-cache(LLC) miss
rate. Other related works, such as [2][3][4], have the similar
idea that deduce power consumption from resource usage.
 All previous methods are defective in the situation that the
hyper-threading of host processor is enabled, where the
relationship between power consumption and CPU utilization
changes when the hyper-threading function takes effect. We
give a solution to this issue. Besides, the power consumption of
cooling fans is separated outside the static power in traditional
metering methods.
Power capping are usually achieved by load balancing at
the application level, or by degradation at hardware level. VM
In previous methods, the power consumption of cooling
fans is included in the static power of host, and is always
shared averagely by all VMs currently running at the host.
However, fan speed is always varying due to varying
environment, which results in a varying power consumption, as
shown in Fig. 3. Further more, average sharing of fan power is
not fair, because the fan power is consumed for cooling while
not all VMs produce the same heat.
&388WLOL]DWLRQ '\QDPLF3RZHU

power provisioning and capping were discussed in [3], and  

'\QDPLF3RZHU:
analysis based on workload traces shown that distinct power  
 

&38
saving could be achieved. As a major step forward to make this
 
practical, we identify several key issues, define corresponding  
strategies, and design and implement an prototype system  
finally, which gives a good demonstrate how VM power  
metering could play a part in power capping.      
7RWDO&RXQWRI90

III. VIRTUAL MACHINE POWER METERING Fig. 2. Different dynamic power per VM at the same CPU utilization.
The energy consumption of a computer could be divided

into two parts: static power and dynamic power. The static
power is consumed whenever a computer is powered on, and 

+RVW3RZHU:
the dynamic power is consumed when a computer executes 
instructions, accesses memory or performs other operations. 


A. Challenges

Dynamic power metering when hyper-threading enabled in
          
host processor. )DQ6SHHGNUSP

When hyper-threading of host processor is enabled, the

dynamic power of host is not linear with CPU utilization all the Fig. 3. Affect of varying fan speed to host power consumption.
time, as shown in Fig. 1. There are 6 VMs with the same
configuration on a host which has 24 logical cores. Each of the B. Methodology
VMs has 4 logical cores, 8GB memory. All VMs run the same
As discussed in the previous section, in this improved VM
benchmark, 249.mcf in SPEC CPU2006, and keep the same
machine power metering method, we separate the fan power
CPU utilization (400%) of host. When the number of running
from static power and analysis it independently. Specifically,
VMs is above 3, i.e. their total CPU utilization is more than
The static power could be divided into two parts: constant
1200%, the hyper-threading of host CPU takes effect, as a
power and fan power as shown in equation (1). All VMs share
result, the dynamic power does not increase much as the CPU
the constant power of the host where the VMs run on averagely
utilization grows.
as shown in equation (2).

PVM ( i ) _ static PVM ( i ) _ const  PVM ( i ) _ fan (1)

'\QDPLF3RZHU:




PVM ( i ) _ const PHost _ const / N VM (2)

To address the challenge that inferring fan power from fan
 speed, we construct a one dimensional lookup table (LUT) for
 each host which gives an empirical power value at certain fan
      
speed rate as shown in equation (3). The LUT for a host,
&388WLOL]DWLRQ
LHost_fan, is constructed when any application of the host has not
been deployed. All hosts that have the same configuration use
Fig. 1. Host dynamic power at different CPU utilization.
the same lookup table. All VMs shares the fan power of the
host where the VMs run on according to the dynamic power of
The traditional methods of dynamic power metering does each VM, as shown in equation (4).
not apply, which deduce power consumption from resource
utilization. As shown in Fig. 2, the dynamic power per VM PHost _ fan LHost _ fan ( S Host _ fan ) (3)
becomes less when the number of running VMs increases.
Varying power of fans
 PVM ( i ) _ dynamic
PVM ( i ) _ fan PHost _ fan u NVM
¦ PVM ( n ) _ dynamic
n 1
We get the overall power of a host at different fan speed
under the condition of no-load and find the overall power at the
highest fan speed of the host. The highest power of fan can be
known by fan product manuals. At this moment, we can get
the constant power of the host by subtracting the highest power
of fan from the overall power of the host at the highest fan
speed. In addition, fan power of the host at different fan speed
can also be got by subtracting the constant power from the
overall power of the host at different fan speed under the
condition of no-load. In this way, we can construct a one
dimensional lookup table (LUT) for each host.
Under the condition of the hyper-threading function being
opened, the utilization rate of processor is no longer linear with
the power consumption of it. The existing measuring methods
(including the method we have proposed before) is no longer
valid. In this situation, all VMs can share the estimated
dynamic power of the host in accordance with a kind of
proportion as shown in the equation (5).
P 'VM ( i ) _ dynamic
PVM ( i ) _ dynamic PHost _ dynamic u NVM
¦P
n 1
'
VM ( n ) _ dynamic
where, P'VM(i)_dynamic is the original VM dynamic power given
by traditional method, and PVM(i)_dynamic is the final value
expected.
IV. APPLICATIONS
A. Power Capping
Power capping is method to limit the amount of electricity
that servers can consume at any given time. The cap is a
definitive limit that the server will not exceed, regardless of its
current workload. The cap has no effect until the server reaches
its power consumption limit. At that point, measures have to be
taken to reduce the power consumption. In an virtualized
environment, one of these measures is to schedule (migrate)
one or more virtual machines out from the host server.
One common presentation of power capping is shown in
equation (6). The minimum power (Pmin) and maximum power
(Pmax) are pre-defined for each host according to its hardware
configuration, and a capping percentage (C%) is set or adjusted
according to power limit. The unit of C is 1%, and the range is
0ӊC<100.
Pcap Pmin  ( Pmax  Pmin ) u C %
Schedule strategy
(1) Detect hosts near capping
We can use the following inequality to detect the host
whose power is close to the limit. T is set as the detection
threshold. The unit of T is 1%, and the range is 0ӊTӊC. The
condition of determining whether the current power of a host
(4) (P) is close to the limit is as follows:
P t Pth Pmin  ( Pmax  Pmin ) u T % (7)
(2) Determine which VM to migrate out
When the overall power of a host at the current time meets
or exceeds the detection threshold, a VM on the host should be
migrated from the host to some other host so as to ensure
power capping. We select the VM that has the minimum
dynamic power and after migrating which, we can make the
power of the host reduced below the threshold value. In this
way, we can select the VM which should be migrated.
The reason of selecting the VM with the minimum dynamic
power to migrate is that it is the fastest to migrate. As shown in
Fig. 4, for the VMs that have the same configuration (having
same number of logical cores and same memory capacity), the
smaller the dynamic power of a VM, the shorter the time
needed for migration.

/LYH0LJUDWLRQ7LPHV




(5) 


    
90'\QDPLF3RZHU:
Fig. 4. Live migration time at different dymaic power comsumption.
(3) Determine where (which host) to migrate the VM
We choose the host which has the largest idle power as the
destination host where the VM should be migrated. Idle power
refers to the subtraction between the current power and the
threshold power as shown in the equation below.
Pfree Pth  P (8)
B. Status Monitoring
To monitor a datacenter, we can gather status information
of all hosts and VM. These status contains power consumption
and resource utilization, including CPU utilization, memory
utilization, cache missing rate.
V. EVALUATIONS
A. Prototype System
We build a prototype system to demonstrate the virtual
(6) machine power metering and its applications. The system
architecture is shown in Fig. 5. The VPower at each host is
used for VM and host power metering. The GInfor at an
external server is used for gathering status information to
Ganglia, an open and free cluster monitor system. Besides, the
VScheduler at this server control the VM live migration based
on power consumption status of all VMs and hosts, to achieve
power capping. Further more, the WebUI gives an interface to
user, containing real time status table, power bars, and
historical trends of all metrics.
B. Experimental Setup
The prototype system is tested on three HUAWEI Tecal
RH2285 rack servers (as shown in Fig. 6) with two Intel
Xeon(R) E5645 6-core processors with hyper-threading
enabled, resulting in 24 logical cores. Each server includes
64GB RAM and one 1TB hard disk, which is for the host OS
D. Status Monitoring
In the WebUI of prototype system, as shown in Fig. 5, real
time status information including power consumption and
resource utilization of all VMs are listed in a table, VM powers
sorted by the VM's locating host are shown in a bar chart, and
historical trends of all metrics are given by Ganglia.
YP YP[ YP


90'\QDPLF3RZHU

CentOS 6.2 x86_64 with KVM hypervisor. One of the three 

servers as the scheduler and the Ganglia central node, the other 
tow serve as host machine, on which six VMs (vm- 
101/102/103/108/109/110) are placed. Besides, another 

machine (not shown) with four 1TB disks acts as an NFS           
Server which is used to store VM images. 7LPHV

Fig. 7. Dynamic power characteristic of VMs.



+RVW3RZHU:







              
7LPHV

KZ KZ WKUHVKROG FDS

Fig. 8. Migrate VM to power capping.

VI. CONCLUTION
Fig. 5. Architecture of prototype system.
We proposed an improved metering method for VM power
consumption and presented scheduling strategies for doing live
migration of VMs based on power consumption to achieve
power capping. A prototype system was introduced and
experiments were carried out. Experimental result shows that
our method gives accurate power consumption of VMs and
scheduling based on power consumption is able to achieve
power capping effectively.

Fig. 6. Experimental setup..

All VMs have the same configuration: 4 logical cores and
8GB memory. The dynamic power characteristic (implying the
workload feature) of all VMs is shown in Fig. 7. Both vm-101
and vm-110 have the same periodical work load with the
period of 600s and with the dynamic power ranging from 0W
to about 60W. However, work load at vm-110 starts 400s later
than that at vm-101. Other VMs are all running the same
constant workload with the dynamic power around 12W.
C. Power Capping
In our prototype system, the power cap and threshold of
each host are 227.5W and 260.6W, respectively. As shown in
Fig. 8, the VSheduler is able to detect in time hosts with power
near the threshold, choose proper VMs to migrate out, and
make the host power under cap finally.
ACKNOWLEDGMENT
This work was supported by the Strategic Priority Research
Program of the Chinese Academy of Sciences under grant
No.XDA06010401, National Science Foundation of China
under grant No. 61100010, No. 61202057 and No. 61303055.
REFERENCES
[1] Zhixiong Jiang, Chunyang Lu, Yushan Cai, Zhiying Jiang and Chongya
Ma: "VPower: Metering Power Consumption of VM", in 4th IEEE
International Conference on Software Engineering and Service Science
(ICSESS), May 2013, pp. 483 - 486.
[2] A. Bohra and V. Chaudhary, "VMeter: Power modelling for virtualized
clouds," in Parallel Distributed Processing, Workshops and Phd Forum
(IPDPSW), 2010 IEEE International Symposium on, april 2010, pp. 1-8.
[3] Aman Kansal, Feng Zhao, Jie Liu, Nupur Kothari, Arka A. Bhattacharya,
"Virtual machine power metering and provisioning," in Proceedings of
the 1st ACM Symposium on Cloud Computing, pp. 39-50, 2010
[4] Bhavani Krishnan, Hrishikesh Amur, Ada Gavrilovska, Karsten Schwan:
"VM power metering: feasibility and challenges", SIGMETRICS
Performance Evaluation Review 38(3): 56-60 (2010)