Real Time Power Estimation and Thread Scheduling via Performance Counters
Karan Singh Major Bhadauria
Computer Systems Laboratory
Cornell University
Ithaca, NY, USA
{karan major}@csl.cornell.edu
Abstract
Estimating power consumption is critical for hardware
and software developers, and of the latter, particularly for
OS programmers writing process schedulers. However, ob-
taining processor and system power consumption informa-
tion can be non-trivial. Simulators are time consuming and
prone to error. Power meters report whole-system consump-
tion, but cannot give per-processor or per-thread informa-
tion. More intrusive hardware instrumentation is possible,
but such solutions are usually employed while designing the
system, and are not meant for customer use.
Given these difficulties, plus the current availability of
some form of performance counters on virtually all plat-
forms (even though such counters were initially designed
for system bring-up, and not intended for general program-
mer consumption), we analytically derive functions for real-
time estimation of processor and system power consump-
tion using performance counter data on real hardware. Our
model uses data gathered from microbenchmarks that cap-
ture potential application behavior. The model is indepen-
dent of our test benchmarks, and thus we expect it to be well
suited for future applications. We target chip multiproces-
sors, analyzing effects of shared resources and temperature
on power estimation, leveraging our model to implement a
simple, power-aware thread scheduler. The NAS and SPEC-
OMP benchmarks shows a median error of 5.8% and 3.9%,
respectively. SPEC 2006 shows a marginally higher median
error of 7.2%.
1 Introduction
Power and thermal constraints limit processor frequency.
In response, computer architects have shifted to chip mul-
tiprocessors (CMPs) to retain performance improvements
without increasing power envelopes, thus such designs trade
higher frequencies and voltages for more cores. When opti-
ACM SIGARCH Computer Architecture News
Sally A. McKee
Department of Computer Science and Engineering
Chalmers University of Technology
Göteborg, Sweden
mckee@chalmers.se
mizing software or evaluating new architectures for existing
software, energy efficiency is now a critical part of perfor-
mance analysis. If the OS is aware of the power consump-
tion of various processes within the system, it can prior-
itize processes based on thermal constraints and available
remaining power, or schedule processes to remain within a
given power envelope. Unfortunately, it is very hard to ex-
pose run-time power consumption of a given processor to
the software developer, computer architect, or OS.
Power meters retrieve total system and CMP power con-
sumption. Although existing hardware can be modified to
monitor the current and power draw of any CPU socket, per
core power consumption is difficult to measure with present
multicore designs, because the processors share the same
power planes. Embedding measurement devices on chip is
not financially feasible. Furthermore, the accuracy of the
power meter can can be affected when the current draw
rapidly fluctuates with program phases.
We propose using Performance Monitoring Counters
(PMCs) to estimate power consumption of any processor
via analytic models. Performance counters on chip are gen-
erally accurate [15] (if used correctly), and they provide sig-
nificant insight into the processor performance at the clock-
cycle granularity. PMCs are already incorporated into and
exposed to user space on most modern architectures. Accu-
rately estimating real-time power consumption enables the
OS to make better real-time scheduling decisions, adminis-
trators to accurately estimate the maximum number of us-
able threads for data centers, and simulators to accurately
estimate power without actually simulating it. Addition-
ally, a power meter is not required per system. Our ana-
lytic model can be queried on multiple systems regardless
of the programs or inputs used. This is possible because
our model uses microbenchmark data independent of pro-
gram behavior. We write these microbenchmarks to gather
PMC data that contribute to the power function. We use
these data to form our power model equations. We thus es-
timate power for single threaded and multithreaded bench-
mark suites, and make the following contributions:
46 Vol. 37, No. 2, May 2009
 FP Units DISPATCHED FPU:ALL
0.23 RETIRED MMX AND FP INSTRUCTIONS:ALL
Inst Retired RETIRED BRANCH INSTRUCTIONS:ALL
RETIRED MISPREDICTED BRANCH INSTRUCTIONS:ALL
RETIRED INSTRUCTIONS
0.39 RETIRED UOPS
Stalls DECODER EMPTY
0.2 DISPATCH STALLS
Memory DRAM ACCESSES PAGE:ALL
DATA CACHE MISSES
L3 CACHE MISSES:ALL
MEMORY CONTROLLER REQUESTS:ALL
0.33 L2 CACHE MISS:ALL

Table 1. PMCs Categorized By Architecture and Ordered

(Increasing) by Correlation (Based on SPEC-OMP Data)

100
90
80

% miss rate
Figure 1. AMD Phenom Annotated Die [1] 70
60
50
40
• We achieve accurate per-core estimates of multi- 30
threaded and multiprogrammed workloads on a CMP 20
10
with shared resources (an L3 cache, memory con- 0
troller, memory channel, and communication buses). caAD s
lc M
s ea x
gr FD lII
om TD

lib les lb s
qu lie m
an 3d
m
m cf
omna ilc
n md
vr p
spsjen y
hi g
t x3
o
ze cbmrf
us k
p
us ve

em d uli

ac

a
poetp

xa ont

m
n w
m
tu

n
• We observe and quantify the interacting effects of
ct a
ca bw

la
power and temperature.
G

• We achieve real-time power estimation, without the

need for off-line benchmark profiling.
• We illustrate an application of our real-time predic-
tions for making thread scheduling decisions at run-
time.
We achieve median errors of 5.8%, 3.9%, and 7.2% for
the NAS, SPEC-OMP, and SPEC 2006 benchmark suites,
respectively. We leverage our real-time estimation to dy-
namically schedule workloads, achieving a reduced power
envelope through suspension of appropriate threads.
2 Methodology
We separate the AMD Phenom performance counters
into four categories: FP Units, Memory, Stalls, and
Instructions Retired. We gather insight into these
categories by examining the processor die in Figure 1.
The shared L3 and private L2 caches take up significant
area, as does the floating point units (FPUs) and front end
logic. Tracking retired instructions gives good perspective
on overall processor performance, and categorizing instruc-
tions into FP or INT indicates what units the instruction mix
uses. Monitoring L2 misses allows us to track use of the
L3 caches, since L2 cache misses often result in L3 cache
misses, which then lead to off-chip memory accesses. Fig-
ure 2 graphs L3 misses normalized to total L3 cache ac-
Figure 2. L3 Cache Miss Rates for SPEC 2006
cesses for SPEC 2006 benchmarks using one reference in-
put. We find that the L3 cache has a large miss rate, since it
is essentially a non-inclusive victim cache.
Another architectural feature responsible for power con-
sumption in high performance processors is the out-of-order
logic. While no single PMC tracks out-of-order logic use,
the number of cycles stalled due to resource limits (DIS-
PATCH STALLS) indicates total stalls due to branches, full
load/store queues, reorder buffers, and reservation stations.
This PMC represents how often the CPU stalls, which pro-
vides insight into stalled issue logic, indicating potentially
less power consumption. However, if the source of these
stalls is the reservation station or reorder buffer, then the
CPU is trying to extract instruction level parallelism, in-
dicating /em increased power consumption. For example,
if all instructions fetched can subsequently be issued and
dispatched, the reservation stations and reorder buffers will
remain a constant size (as instructions are retired in or-
der). Should a fetched instruction stall, out-of-order logic
attempts to execute another, requiring dynamic power ex-
penditure, since more reservation stations are examined to
check for a new instruction’s dependences being satisfied.

ACM SIGARCH Computer Architecture News 47 Vol. 37, No. 2, May 2009
2.1 Event Selection
We split the AMD Phenom PMCs into the four cat-
egories chosen above: FP Units, Memory, Stalls,
and Instructions Retired. We use a Watts Up Pro
power meter and pfmon to gather power and performance
data for 13 counters from the entire SPEC-OMP benchmark
suite. The specific categories and the PMCs that fall within
them are shown in Table 1. The PMCs in each category are
in increasing order of correlation with power. We choose
Spearman’s rank correlation to assess the relationship be-
tween the counter value and power. Compared to vanilla
correlation, Spearman’s rank correlation does not require
assumptions about the frequency distributions of the vari-
ables. After performing this rank correlation on the data, we
select the top counter from each bucket. The AMD Phenom
only tracks four counters simultaneously, thus our choice of
four categories for the PMCs works well on this platform.
Since our power predictor is designed to work in real-time,
we are bound by this four-counter limit. However, this is an
architecture-specific characteristic that varies by platform.
The performance counters we use are:
• e1 : L2 CACHE MISS:ALL
• e2 : RETIRED UOPS
• e3 : RETIRED MMX AND FP INSTRUCTIONS:ALL
• e4 : DISPATCH STALLS
We claim that data from these four counters capture
enough information to predict core and system power. This
claim is backed by results in Section 4. Next, we discuss the
microbenchmarks that exercise our chosen counters, and we
then form the model based on the data collected.
2.2 Microbenchmarks
Since we wish our model to be independent of appli-
cations used, we write microbenchmarks to stress the four
counters selected, and we explore the space of their cross
product. This space spans values from zero to several bil-
lion, with large variations in values among benchmarks. For
example, integer benchmarks have few floating point op-
erations, and some computationally intensive benchmarks
have few L2 or L3 cache misses. We try to cover the ex-
treme boundaries along with common cases, but our mi-
crobenchmarks cannot cover all regions. A more thorough
set of benchmarks could improve estimation, but would in-
crease training time. We only target three dimensions of the
space, since one of our PMCs (DISPATCH STALLS) can-
not be explicitly targeted with code. Our benchmark codes
consists of a large for loop and a case statement that it-
erates through different code nests, depending on the loop
index. Our synthetic benchmarks consists mainly of a set
of assign statements (moves) and arithmetic/floating point
operations, so we compile with no optimizations to prevent
the compiler from removing redundant lines of code.
ACM SIGARCH Computer Architecture News
We run four copies of the microbenchmarks simultane-
ously. We isolate the data for any single CPU, since all cores
exhibit the same PMC data. To ensure none of the separate
threads are out of sync with each other, we run each pro-
gram phase separately. We use the SPEC-OMP benchmark
suite for selecting appropriate counters only. Our code is
generic, and does not use any code from benchmarks used
for testing (SPEC 2006, SPEC-OMP and NAS). The mi-
crobenchmarks explore the space spanned by the four se-
lected counters. This means future application behavior
should fall within the same space, and allows us to claim
that our approach works independently of the benchmark
suite. We verify this by testing our model on the NAS and
SPEC 2006 benchmark suites, applications that we do not
use during model building. Since this is an empirical pro-
cess, verification lies in the quality of predictions.
2.3 Forming the Model
For the model, we use only data from our microbench-
marks. We normalize collected performance counter values
to the elapsed cycle counts, yielding an event rate, ri ,
per counter. The prediction model uses these rates as input.
We model per-core power for each thread using our piece-
wise model based on multiple linear regression. We collect
PMC values every second while running the microbench-
marks, and generate the model based on this data. Our ap-
proach produces the following function (Equation 1), map-
ping observed event rates ri to core power Pcore :

F1 (g1 (r1 ), ..., gn (rn )), if condition
P̂core = (1)
F2 (g1 (r1 ), ..., gn (rn )), else
where ri = ei /(cycle count)
Fn = p0 + p1 ∗ g1 (r1 ) + ... + p2 ∗ gn (rn ) (2)
The function consists of linear weights on transforma-
tions of event rates (Equation 2). Spearman’s rank cor-
relation operates independently of frequency distributions,
thus transformations can be linear, inverse, logarithmic, ex-
ponential, or square root (whatever makes the data more
amenable to linear regression, which helps prediction accu-
racy). We choose a piece-wise linear function because we
find code to behave significantly differently for low PMC
values. The approach lets us capture more detail about the
core power without sacrificing the simplicity and ease of
linear regression. For example, were we to form a model
for the data in Figure 3, we would find that neither a linear
nor exponential transformation fits the data. However, were
we to break the data into two parts, we would find a piece-
wise combination of the two fits much better, as in Figure 4.
We use least squares to determine weights for function
parameters. Each part of the piece-wise function can be
48 Vol. 37, No. 2, May 2009
 data
linear function
exponential function
F(x)
x ues. Benchmarks bt, lu and namd are run across all four
Figure 3. An Illustrative Example of Best-Fit Continuous
Approximation Functions
data
piecewise function
F(x)
x
Figure 4. An Illustrative Example of a Better Fitting
Piece-Wise Function
written as a linear combination of transformed event rates
(Equation 2), and is easily solved using a least squares es-
timator as in Contreras et al. [7]. We obtain the piece-wise
linear model shown in Figure 5. We find the function be-
havior to be significantly different for very low values of
the L2 cache miss counter compared to the rest of the space.
We break our function based on this counter. Since the L3
is non-inclusive, most L2 cache misses trigger off-chip ac-
cesses contributing to total power. We also observe that the
power grows with increasing retired uops, since the CPU
is doing more work. All counters have positive correla-
tion with power, except for the retired FP/MMX instructions
PMC. This is expected, since such instructions have higher
latencies; this class of instructions reduces the throughput
of the system, resulting in lower power use. Finally, the
dispatch stalls PMC correlates positively with power. This
can be due to reservation stations or reorder buffer dispatch
stalls, where the processor attempts to extract higher de-
grees of instruction level parallelism (ILP) from the code.
Dynamic power increases from this logic overhead.
2.4 Temperature Effects
We are interested in the effect of temperature on system
power. Ideally, power consumption does not increase over
ACM SIGARCH Computer Architecture News
time. However, since static power is a function of volt-
age, process technology and temperature, increasing tem-
perature leads to increasing leakage power, and adds to total
power. We concurrently monitor the temperature and power
of the CMP, to see their relationship. Figures 6 graphs
temperature (in Celsius) and power consumption (in Watts)
over time. Results are normalized to their steady-state val-
cores of the CMP, with results capped at 120 seconds. For
namd, four instances are run concurrently since it is single-
threaded. Performance counters and program source code
are examined to insure the work performed is constant over
time. The programs exhibit varying increases in power and
temperature over time. Clearly, temperature and power af-
fect each other. Not accounting for temperature could lead
to increased error in power estimates. However, not all sys-
tems support temperature sensors on die, or per-core. We
omit temperature readings in this model since our hardware
lacks support for per-core temperature sensors.
3 Experimental Setup
We evaluate our work using the SPEC 2006 [14], SPEC-
OMP [2], and NAS [4] benchmark suites. All are com-
piled for the AMD x86 64 64-bit architecture using gcc
4.2 with -O3 optimization flags (and -OpenMP where ap-
plicable). We use reference input sets for SPEC 2006 and
SPEC-OMP, and the medium input set (B) for NAS. All
benchmarks are run to completion. We use the pfmon util-
ity from the perfmon2 library to access hardware perfor-
mance counters from user space, and run on Linux kernel
version 2.6.25. Table 2 details the system configuration.
System power is based on the processors’ being idle and
powered at their lowest frequency and voltage. All power
measurements are based on current draw from the outlet
by the power supply. We use a Watts Up Pro power meter
to measure power consumption (accurate to within 0.1W).
The meter updates its power values every second. Base-
line power for the system is 85.1W running at 2.2 GHz,
and 76.8W at 1.1 GHz. For the purpose of model forma-
tion only, we assume the base power without processor to
be 68.5W. This simplifying assumption aids in faster model
formation without the need for more complicated measur-
ing techniques. We calculate per-core power by subtracting
the base power and dividing by four.
Some performance counters measure shared resources,
which gives statistics across the CMP and not per core.
Some performance counters could be further subdivided by
type. For example, cache and DRAM accesses can be bro-
ken down into cache or page hits and misses, while dis-
patch stalls can be broken down by branch flushes, or full
queues (reservation stations, reorder buffers, floating point
units). The AMD processor in this work can only sam-
49 Vol. 37, No. 2, May 2009
 
1.15 + 65.88 ∗ r1 + 11.49 ∗ r2 + −3.356 ∗ r3 + 17.75 ∗ r4 , r1 < 10−5
P̂core = (3)
23.34 + 0.93 ∗ log(r1 ) + 10.39 ∗ r2 + −6.425 ∗ r3 + 6.43 ∗ log(r4 ), r1 ≥ 10−5
where ri = ei /2200000000

Figure 5. Piece-wise Linear Function for Core Power Derived from Microbenchmark Data (1 Second = 2.2 Billion Cycles)

Normalized to Steady State

Normalized to Steady State
Normalized to Steady State

1.10 1.10 1.10

1.05 1.05 1.05
1.00 1.00 1.00
0.95 0.95 0.95
0.90 temperature 0.90 temperature 0.90 temperature
0.85 power 0.85 power 0.85 power
0.80 0.80 0.80
0 20 40 60 80 100 120 0 20 40 60 80 100 120 0 20 40 60 80 100 120
(a) bt (b) namd (c) lu

Figure 6. Power vs. Temperature on Our 4-Core CMP

Frequency
Process Technology
Processor
Number of Cores
L1 (Instruction) Size
L1 (Data) Size
L2 Cache Size (Private)
L3 Cache Size (Shared)
Memory Controller
Memory Width
Memory Channels
Main Memory
Table 2. CMP Machine Configuration Parameters
ple four counters simultaneously, which is why we narrow
our design space to using only four performance counters.
For power-aware process scheduling, we suspend processes
determined to exceed the available power envelope. Sus-
pended processes are later restored, and they continue exe-
cuting when sufficient power becomes available.
4 Evaluation
We evaluate the accuracy of our power model using sin-
gle and multi-threaded benchmarks, using the entire CMP
to test our results. We later leverage this real-time estima-
tion to make power-aware scheduling decisions, suspending
processes to maintain a given power envelope. Note that the
scheduler is a user-level application, and is quite simple –
much more sophisticated schedulers (e.g., taking priorities
into consideration) are possible, but ours serves as a proof
of concept.
2.2 GHz (max), 1.1 GHz (min) 4.1 Evaluating Power Estimation
65nm SOI
AMD Phenom 9500 CMP
4 We test our derived power model by comparing mea-
64 KB 2-Way Set Associative sured to predicted power in Figures 7(a), 7(b), 7(c) for
64KB 2-Way Set Associative
512 KB/core 8-Way Set Associative
NAS, SPEC-OMP, and SPEC 2006, respectively. Each
2 MB 32-Way Set Associative multi-threaded benchmark is run across the entire CMP, and
Integrated On-Chip multiple copies are spawned for single-threaded programs.
64-bits /channel
Data reported are per core. Our estimation model tracks
2
4 GB PC2 6400(DDR2-800) power consumption for each benchmark fairly well. We
find a large difference between estimated and actual power
for some benchmarks, such as bt and ft. This can partially
be attributed to lack of dynamic temperature data in our es-
timation model. Actual power values range from 19.6W for
ep to 26.6W for mgrid for a variation over 35%.
Figures 8(a), 8(b), and 8(c) show percentage error for
each suite. We attribute error in power estimates to tempera-
ture, the limited number of PMCs simultaneously available,
and parts of the counter space possibly unexplored by our
microbenchmarks. For instance, temperature can increase
power consumption by up to 10%. NAS and SPEC-OMP
have average median error of 5.8% and 3.9%, respectively.
SPEC 2006 has slightly higher median error of 7.2%.
Figure 9 shows the Cumulative Distribution Function
(CDF) for all three benchmark suites taken together. This
gives us a picture of the coverage of our model. For exam-
ple, 85% of predictions across all benchmarks have less than
10% error. The CDF helps illustrate the model’s fits, show-
ing that most predictions have very small error. To further
test model robustness, we examine system power estimates
for a multiprogrammed workload. Figure 11(d) shows sys-
tem power consumption for ep (NAS), art (SPEC-OMP),
mcf and hmmer (SPEC 2006), run concurrently. We sum
per-core power to estimate system power. Our model con-

ACM SIGARCH Computer Architecture News 50 Vol. 37, No. 2, May 2009
 % Median Error Power (W)

0
10
20
30

0
5
10
15
20
bt
% Median Error Power (W)
bt cg

0
10
20
30

0
5
10
15
20
as
ta
as cg bz r
ta ep
b ip
actual

bz r ep ca wa 2
predicted

bw ip2 ct ve
ca a us s ft
ct ve AD
us s ft ca M

actual
AD lc lu
ca M ul

predicted
(a) NAS

(a) NAS
lc G ix
ul lu em de lu
G ix -h
em de lu sF alll p
sF alll
-h
p
D
D ga TD
m m

ACM SIGARCH Computer Architecture News

ga TD m e g
m g go ss
e
go ss gr bm
sp om k sp
gr bm
om k a
h2 cs
h2 acs ua 64 ua
64 hm ref
hm ref m
m % Median Error er Power (W)

51
er l am
0
10
20
30

0
5
10
15
20
le bm m
(c) SPEC 2006

(c) SPEC 2006

le lbm lib slie p
lib sli am qu 3 ap
qu e3 m an d
an d p tu pl
u
tu ap m
m pl
u m ap
m cf si
cf
actual

ap
om i m si om mil
c ar
lc
predicted

Figure 7. Measured vs. Predicted Power

pe net t
pe net ar rlb pp ga
rlb pp t en fo
en ga rt
fo fm

Figure 8. Median Errors for Given Benchmark Suites

po ch rt po ch
vr fm vr a3
a a3
ay d
xa sop y d xa sop m
(b) SPEC-OMP

(b) SPEC-OMP
la le m la lex
nc gr
nc x gr b id
b id qu
ze mk qu ze mk
us us ak
m ak m e
p e p
sw sw
w i m w im
up up
w w
is is
e e

Vol. 37, No. 2, May 2009

 Fraction of Space Covered

  

1.0

0.8         

     
        
0.6

0.4                

0.2
 
  
  
   
  
  
   
  
  
   
  
  
 
0.0
0 10 20 30
  
    
% Error
Figure 9. Cumulative Distribution Function (CDF) Plot
Showing Fraction of Space Predicted (y-axis) under a Given
Error (x-axis)
servatively over-estimates power consumption. After ep
finishes executing, power prediction decreases with mea-
sured system power. System power further decreases after
art completes, and only two cores continue. Even with the
interaction of shared resources, our model accurately tracks
CMP system power consumption.
4.2 Power-Aware Thread Scheduling
We use the power predicted for processes to schedule
them within a multi-programmed workload on the CMP. We
suspend processes to remain below the system power enve-
lope. For these experiments, we assume the system power
envelope to be degraded by 10, 20, or 30%.
We develop a user-space scheduler in C that spawns a
process on each of the four cores of the AMD Phenom,
monitors their behavior via pfmon, and schedules processes
to run within a fixed power envelope. Figure 10 illustrates
its setup and use. The program makes real-time predictions
for per-core and system power based on collected perfor-
mance counters, and suspends processes as the power enve-
lope is breached. It suspends the processes such that system
power is just below the power envelope. For example, as-
sume that current system power is 190W and the power en-
velope is 180W. For simplicity, we have to choose between
two processes consuming 20W and 25W respectively. The
scheduler suspends the first process to bring system power
down to 170W, rather than choosing the second process and
being further away from the envelope (at 165W).
We present examples of running a randomly chosen
workload at 180W, 160W, and 140W, which represent ap-
proximately 90%, 80%, and 70% of maximum power us-
age when no processes are suspended. For fairness, we
run multi-threaded programs with one thread only. Since
if we were to suspend a multi-threaded process, it would
span multiple cores and would be far below the power en-
velope. Benchmarks are chosen arbitrarily, consisting of
ACM SIGARCH Computer Architecture News
Figure 10. Scheduler Setup and Use
ep from NAS, art from SPEC OMP, and mcf and hmmer
from SPEC CPU 2006. Figure 11(d) shows workload ex-
ecution without a power envelope. In Figure 11(a), there
is a sudden drop in system power during the initial execu-
tion of the workload. The total system power of 177.7W
is far above the applied power envelope of 140W. At this
point, ep, art, mcf, and hmmer are using 19.8W, 20.5W,
21.4W, and 22.4W respectively. The scheduler suspends ep
and art because their sum of 40.3W brings the power just
below the envelope of 140W. The drop in power before the
500 second mark is where the one process ends, and the
scheduler resumes another. Similarly, in Figure 11(c), total
system power of 181.1W predicted is just above the power
envelope of 180W, and ep, art, mcf, and hmmer are us-
ing 19.7W, 24.7W, 20.7W, and 22.2W respectively. The
drop at about 20 seconds into the execution of the work-
load illustrates the scheduler’s suspending ep, since it al-
lows total power to remain closest to the power envelope.
The process is later resumed to complete execution.
Figures 11(a), 11(b), and 11(c) show that the Measured
and Predicted power match up well. We are able to fol-
low the power envelope strictly, and do so entirely on the
basis of our prediction-based scheduler. This obviates the
need for a power meter, and would be an excellent tool for
software-level control of total system power.
5 Related Work
Prior work falls under two different approaches. The first
strategy estimates power consumption based on monitor-
ing functional unit usage [11, 16]. The second strategy de-
rives mathematical correlations between performance coun-
ters and power consumed, independent of the underlying
functional units [7]. In our work, we combination the two
approaches, using correlation and architectural knowledge
to choose appropriate performance counters, and using ana-
lytic techniques to identify the relationship between perfor-
mance counters and power consumption. We briefly outline
related work.
52 Vol. 37, No. 2, May 2009
 200 200

System Power (W)

System Power (W)
150 150

100 100
predicted predicted
50 actual 50 actual
envelope envelope
0 0
0 200 400 600 800 1000 0 200 400 600 800 1000
Time (sec) Time (sec)
(a) 140W (b) 160W
200 200

System Power (W)

System Power (W)

150 150

100 100
predicted predicted
50 actual 50 actual
envelope
0 0
0 200 400 600 800 1000 0 200 400 600 800 1000
Time (sec) Time (sec)
(c) 180W (d) None

Figure 11. Scheduler with Given Power Envelope

Joseph and Martonosi [11] estimate power using per-
formance counters in a simulator (Wattch [6], integrated
with SimpleScalar [3]), and on real hardware (Pentium Pro).
Their power estimates are derived offline after multiple
benchmark runs, since they require more PMCs than are
supported by their hardware. They use PMCs to estimate
power consumption for different architectural resources, but
are unable to estimate power for 24% of the chip, and as-
sume peak power of the structures.
Wu et al. [16] use microbenchmarks to estimate power
consumption of functional units for the Pentium 4 architec-
ture. Dynamic activity for all portions of the chip are not
available, and if they were, the approach would exceed the
limit of four CMPs for run-time power prediction.
Economou et al. [9] use performance counters to pre-
dict power of a blade server. They use custom micro-
benchmarks to profile the system and estimate power for
different hardware components (CPU, memory, and hard
drive), achieving estimates with 10% average error. Like
our work, their kernel benchmarks for deriving their model
are independent of benchmarks used for testing.
Lee and Brooks [12] use statistical inference models to
predict power consumption based on hardware design pa-
rameters. They build correlation based on hardware param-
eters, and use the most significant parameters for training
a model to estimate power consumption. They randomly
sample a large design space and estimate power consump-
tion based on previous power values for the same design
space that is profiled a priori. Unfortunately, this methodol-
ogy requires sampling the same applications for which they
are trying to estimate power consumption. This makes the
approach dependent on having already trained on the appli-
cations of interest. The method is not feasible when execut-
ing programs outside of the sampled design space. At run
time, scheduling is not known a priori, and the behavior of
programs changes depending on their interaction with other
processes sharing the same cache resources.
Curtis-Maury et al. [8] predict optimal concurrency val-
ues using machine learning to estimate power consumption
at various thread counts based on power consumption at the
highest thread count for the NAS benchmark suite. They
require that applications be profiled at one thread count be-
fore predicting power consumption at other thread counts,
making their models application dependent.
Contreras and Martonosi [7] use PMCs on an XScale to
estimate power consumption online at different frequencies.
Like Joseph and Martonosi, they gather data from multiple
PMCs by running benchmarks several times, and join sam-
pled data from different runs together to derive better esti-
mates. This methodology is not feasible at run-time, since it
requires three times th PMCs as can be sampled in real time.
Thus it cannot be used by an OS scheduler or application de-
veloper executing multiple programs in parallel at run time.
Their parameterized linear model does not apply here, as
their work for an in-order single core does not scale to our
platform. This is due to the increased complexity of our
multicore, out-of-order, power-efficient, high-performance
platform.
Merkel and Bellosa [13] use performance counters to es-
timate power per processor in a 8-way symmetric multipro-
cessing (SMP) system, and shuffle processes to reduce over-
heating of individual processors. Bautista et al. [5] sched-
ule processes on a multicore system for real-time applica-
tions. They assume static power consumption for all appli-

ACM SIGARCH Computer Architecture News 53 Vol. 37, No. 2, May 2009
cations. They leverage chip-wide DVFS to reduce energy
when there is slack in the application. Isci et al. [10] ana-
lyze power management policies for a given power budget.
They leverage per-core and domain-wide DVFS to retain
performance while remaining within a power budget. Un-
like their work, we do not require on-core current sensors,
but instead use real hardware to evaluate our work. While
their work involves core adaptation policies, we examine
OS scheduling decisions with fixed cores. We will expand
our approach when per-core DVFS becomes available.
We estimate power consumption for an aggressively
power efficient high performance platform. Our platform
differs with previous literature due to: resources being
shared across cores, performance counters that do not indi-
vidually exhibit high correlation with power, and significant
power variation across benchmarks, depending on work-
load. Additionally, since we desire real-time power esti-
mation, we are confined to using only the number of perfor-
mance counters available at run-time in a single run. This
prevents us from making fair comparisons of this work with
previous work. Our framework can estimate power con-
sumption on new programs that we have not run before,
and evaluates results on a multi-core system, providing a
feasible method of estimating power consumption per-core.
We leverage this to perform real-time power-aware per-core
scheduling on a CMP, which was not previously possible.
6 Conclusions
We derive an analytic, piece-wise linear power model
that maps performance counters to power consumption ac-
curately, independently of program behavior for the SPEC
2006, SPEC-OMP and NAS benchmark suites. We use cus-
tom microbenchmarks to generate data for creating our es-
timation model. The microbenchmarks stress four coun-
ters selected based on their correlation with measured power
consumption values.
We leverage our power model to perform run-time,
power-aware thread scheduling. We suspend and resume
processes based on power consumption, ensuring that a
given power envelope is not exceeded. Our work is likely
useful for consolidated data centers, where virtualization
leads to multiple servers on a single CMP. Using our es-
timation methodology, we can accurately estimate power
consumption at the core granularity, allowing for accu-
rate billing of power usage and cooling costs. Estimating
per-core power consumption is challenging, since some re-
sources are shared across cores (such as caches, the DRAM
memory controller and off-chip memory accesses). Addi-
tionally, current machines only allow us to monitor a few
performance counters per-core, depending on the platform
(Intel, Sun, AMD).
For future work, we will expand this study to schedule
ACM SIGARCH Computer Architecture News
multi-threaded workloads. This adds a layer of complexity
since all the threads of a specific process need to be sus-
pended at once. If only one thread is suspended, it might
be holding locks or other threads might be forced to wait at
barrier points for it. Another strategy is to force the sched-
uler to wait for specific thread spawn points before changing
numbers of threads.
We find temperature to play a significant role in power
consumption. When supported by the hardware, we will in-
corporate thermal data into our estimation model and thread
scheduling strategies. We also wish to investigate the ben-
efits of using a power-aware scheduler for choosing pro-
cessor cores to reduce frequency (DVFS), rather than sus-
pend processes. Much more sophisticated schedulers (e.g.,
with knowledge of priorities or real-time demands) are pos-
sible, and experimenting with these will be interesting to
observe the consequential effects on throughput. As num-
ber of cores and power demands grow, process scheduling
will be critical for efficient computing.
References
[1] AMD Phenom(TM) Quad-Core Processor Die. www.amd.com/us-
en/Corporate/VirtualPressRoom/0,,51 104 572 5731̂50441̃17770,00.html,
Jan. 2008.
[2] V. Aslot and R. Eigenmann. Performance characteristics of the SPEC
OMP2001 benchmarks. In Proceedings of the European Workshop
on OpenMP, Sept. 2001.
[3] T. Austin. Simplescalar 4.0 release note.
http://www.simplescalar.com/.
[4] D. Bailey, T. Harris, W. Saphir, R. Van der Wijngaart, A. Woo, and
M. Yarrow. The NAS parallel benchmarks 2.0. Report NAS-95-020,
NASA Ames Research Center, Dec. 1995.
[5] D. Bautista, J. Sahuquillo, H. Hassan, S. Petit, and J. Duato. A simple
power-aware scheduling for multicore systems when running real-
time applications. In Proc. 22nd IEEE/ACM International Parallel
and Distributed Processing Symposium, pages 1–7, Apr. 2008.
[6] D. Brooks, V. Tiwari, and M. Martonosi. Wattch: A framework
for architectural-level power analysis and optimizations. In Proc.
27th IEEE/ACM International Symposium on Computer Architec-
ture, pages 83–94, June 2000.
[7] G. Contreras and M. Martonosi. Power prediction for intel xs-
cale processors using performance monitoring unit events. In Proc.
IEEE/ACM International Symposium on Low Power Electronics and
Design, pages 221–226, Aug. 2005.
[8] M. Curtis-Maury, K. Singh, S. A. McKee, F. Blagojevic, D. S.
Nikolopoulos, B. R. de Supinski, and M. Schulz. Identifying energy-
efficient concurrency levels using machine learning. In Proc. 1st In-
ternational Workshop on Green Computing, Sept. 2007.
[9] D. Economou, S. Rivoire, C. Kozyrakis, and P. Ranganathan. Full-
system power analysis and modeling for server environments. In
Workshop on Modeling Benchmarking and Simulation (MOBS) at
ISCA, June 2006.
[10] C. Isci, A. Buyuktosunoglu, C.-Y. Cher, P. Bose, and M. Martonosi.
An analysis of efficient multi-core global power management poli-
cies: Maximizing performance for a given power budget. In Proc.
IEEE/ACM 40th Annual International Symposium on Microarchitec-
ture, pages 347–358, Dec. 2006.
54 Vol. 37, No. 2, May 2009
[11] R. Joseph and M. Martonosi. Run-time power estimation in high-
performance microprocessors. In Proc. IEEE/ACM International
Symposium on Low Power Electronics and Design, pages 135–140,
Aug. 2001.
[12] B. Lee and D. Brooks. Accurate and efficient regression modeling
for microarchitectural performance and power prediction. In Proc.
12th ACM Symposium on Architectural Support for Programming
Languages and Operating Systems, pages 185–194, Oct. 2006.
[13] A. Merkel and F. Bellosa. Balancing power consumption in mul-
tiprocessor systems. In EuroSys ’06: Proceedings of the ACM
SIGOPS/EuroSys European Conference on Computer Systems, pages
403–414, Apr. 2006.
[14] Standard Performance Evaluation Corporation. SPEC CPU bench-
mark suite. http://www.specbench.org/osg/cpu2006/, 2006.
[15] V. Weaver and S. A. McKee. Can hardware performance counters
be trusted? In Proc. IEEE International Symposium on Workload
Characterization, Sept. 2008.
[16] W. Wu, L. Jin, and J. Yang. A systematic method for functional
unit power estimation in microprocessors. In Proc. 43rd ACM/IEEE
Design Automation Conference, pages 554–557, July 2006.

ACM SIGARCH Computer Architecture News 55 Vol. 37, No. 2, May 2009