Data Center Efficiency Metrics:

mPUE™, Partial PUE, ERE, DCcE

Dan Azevedo, Symantec
Jud Cooley, Oracle
Michael Patterson, Intel
Mark Blackburn, 1E

Join The Green Grid Technical Forum conversation on Twitter!

As you are tweeting, use #TGG11.
 Presenters
Dan Azevedo, Symantec
Director of Data Center Architecture, Strategy & Innovation
Board Member and Metrics & Measurements Chairman, The Green Grid

Jud Cooley, Oracle

Senior Director, Design for the Data Center
Alternate Board Member, The Green Grid

Michael K. Patterson, Intel

Senior Power & Thermal Architect
Technology & Strategy Chairman, The Green Grid

Mark Blackburn, 1E
Co-Founder & Chief Technologist
Metrics & Measurements Vice Chairman, The Green Grid

Copyright © 2011, The Green Grid

The United States of America, European
Union, and Japan have successfully
completed harmonization efforts around
one data center energy efficiency metric
Power Usage
Effectiveness (PUE)
U.S. Data Center Industry Session
13 Jan 2010

• Attendees:

• Session Objectives:
 Data Center Efficiency Metrics: Develop common definitions for key
metrics and seek consensus on guiding principles for metrics
 Measurement Protocols: Define current status and gaps and coordinate
plans for further development
 Tools: Review status and coordinate development

Copyright © 2011, The Green Grid

U.S. Data Center Industry Session
Taskforce Results

• Released two memos:

 Data Center Industry Leaders Reach Agreement on Guiding Principles
for Energy Efficiency Metrics (Feb 2010)
 Recommendations for Measuring and Reporting Overall Data Center
Efficiency – Version 1 Measuring PUE at Dedicated Data Centers (Jul
2010)

US Consensus Achieved with Public Announcement for PUE with Guidance and
Refinement

• Work in Progress
 Recommendations for Measuring and Reporting Overall Data Center
Efficiency – Version 2 Measuring PUE in a Mixed Use Facility

Copyright © 2011, The Green Grid

Global Metric Harmonization Meetings
26 Mar 2009, 02 Feb 2010, 26 Oct 2010, 15 Feb 2011

• Attendees:
 Ministry of Economy, Trade and Industry
 Green IT Promotional Council
 The U.S. Department of Energy‟s Save Energy Now and Federal Energy
Management Programs
 The U.S. Environmental Protection Agency‟s ENERGY STAR Program
 The European Commission Joint Research Center Data Centers Code of
Conduct
 The Green Grid

• Session Objectives:
 Share global lessons and practices with an objective of arriving at a set of
metrics, indices, and measurement protocols which can be formally endorsed or
adopted by each participant organization to improve data center energy
efficiency globally.

Copyright © 2011, The Green Grid

Global Metric Harmonization Meetings
Taskforce Results

• Released two memos:

 The United States of America, European Union and Japan
Reach Agreement on Guiding Principles for Data Center Energy
Efficiency Metrics (Feb 2010)
 Global Taskforce Reaches Agreement on Measurement
Protocols for PUE – Continues Discussion of Additional Energy
Efficiency Metrics (Feb 2011)

The United States of America, European Union and Japan Successfully Complete
Harmonizing PUE

• Work in Progress:
 Prioritized further areas of work to drive Data Center Energy
Efficiency (see appendix)

8 Copyright © 2011, The Green Grid

Influences Specific to PUE

Global
Harmonization
of Metrics

U.S. Industry Data Center

Taskforce

The Green Grid

Copyright © 2011, The Green Grid

Partial PUE (pPUE)
Rationale for Partial PUE
• PUE can be used to compare data center designs
• Often PUE's are presented that do not take into account all of
the infrastructure components in order to highlight one
particular portion of a data center
– “Container only” PUE
– “Cooling system” PUE
– “Power delivery” PUE
• These comparisons have value, but are not truly PUE
• “Partial PUE” provides a formal language for this
• Partial PUE is abbreviated “pPUE”

Copyright © 2011, The Green Grid

pPUE Definition
• PUE = Total Facility Energy divided by the IT
Equipment Energy
• This takes into account energy use within a facility
• Partial PUE is for energy use within a boundary

pPUE = Total Energy within a boundary divided

by the IT Equipment Energy within that
boundary

Copyright © 2011, The Green Grid

 Example 1

• PUE = 600/475 = 1.26

• pPUE of Container = 500/475 = 1.05

Copyright © 2011, The Green Grid

Example 2

PUE = 2700/2000 = 1.35

PPUE1 = 1700/1500 = 1.13
PPUE2 = 700/500 = 1.40

• Zones 1 and 2 have the same chiller power,

but different IT loads
• So their Partial PUE's are different
Copyright © 2011, The Green Grid
Notations
• N is for “non-IT” power
• I is for “IT” power
• PUE =
(N0 + N1 + N2 + I1 + I2)
(I1 + I2)

(N1 + I1)
pPUE1 =
(I1)
(N2 + I2)
pPUE2 =
(I2)

Zone 0 has no IT, so pPUE is

undefined
Copyright © 2011, The Green Grid
 pPUE Relative to PUE
• Zones 1 and 2 each hold a portion of the IT load:
(I1) (I2)
r1 = r2 =
(I1 + I2) (I1 + I2)

(N0)
• PUE =
(I1 + I2) + (r1 * pPUE1) + (r2 * pPUE2)

(The proof is in the White Paper, note that r1 + r2 = 100%)

The PUE is the overhead seen by all zones plus the Partial
PUE of each zone multiplied by the percentage of that zone's
contribution to the overall IT load

Copyright © 2011, The Green Grid

 What It Means
(N0)
• PUE = + (r1 * pPUE1) + (r2 * pPUE2)
(I1 + I2)
• If one of the pPUE's is bad, work on that
• But if one of the r's is very large, the payback for working on
that is great

Copyright © 2011, The Green Grid

Example 3

This is the same as example 2 except

that Zone 1 is 10 times as large
• PUE = 20,000/15,500 = 1.29
• pPUE1 = 17,000/15,000 = 1.13
• pPUE2 = 700/500 = 1.40
• r1 = 15,000/15,500 = 0.97
• r2 = 500/15,500 = 0.03

Copyright © 2011, The Green Grid

Summary
• Partial PUE can be used to describe the PUE contribution of
a subset of facility components
• Partial PUE refers to components within a boundary
• The boundary can be physical or logical, for example:
– All cooling components
– All power components
– All Tier II components
– All baseline load components
– All departmental components
• PUE is a valuable metric for monitoring and managing data
center power
• If only a portion of the facility is being analyzed, use partial
PUE

Copyright © 2011, The Green Grid

Energy Reuse
Effectiveness (ERE)
“I am re-using waste heat from
my data center on another part
of my site and my PUE is 0.8!”

Copyright © 2011, The Green Grid

 “I am reusing waste heat from my data
center on another part of my site and my
PUE is 0.8!”

• While re-using excess energy from the data center

can be a good thing to do, it should not be rolled into
PUE
• The definition of PUE does not allow this
• There is a metric to do this! ERE

Copyright © 2011, The Green Grid

ERE Definition
Total Energy
PUE =
IT Energy
Cooling + Power + Lighting + IT
PUE =
IT

Total Energy - Reused Energy

ERE =
IT Energy
Cooling + Power + Lighting + IT - Reused
ERE =
IT
Copyright © 2011, The Green Grid
 ERE Alternate Development
Define energy reuse factor (ERF) as: Then:

Reuse Energy ERE = (1 - ERF)  PUE

ERF =
Total Energy
And finally:
Cool + Pwr + Light + IT - Reused
ERE =  (1  ERF)  PUE
IT

ERF and PUE are mathematically related, but differ and

need to be defined and reported clearly.

Copyright © 2011, The Green Grid

 PUE and ERF Ranges
Cooling + Power + Lighting + IT
1  PUE   PUE =
IT

Reuse Energy
0  ERF  1 ERF =
Total Energy

Cool + Pwr + Light + IT - Reused

0  ERE   ERE =
IT

Copyright © 2011, The Green Grid

Data Center Boundary
Consideration
Cooling
and ERF Waste
E

A
Utility
IT F
B

UPS C PDU D

Data Center
A+B Boundary
PUE =
D
Copyright © 2011, The Green Grid
 Use of ERE Non-DC
waste
space

Cool
Absorp
G Chiller
A E
I
Utility
F
ICT
B

UPS C PDU
D

A+B A + B-F Data Center

PUE = ERE = Boundary
D D
Both PUE & ERE valid metrics
Copyright © 2011, The Green Grid
 Improper Use of ERE waste

Cool
AbsorpC
G hiller
A E
I
Utility
F
ICT
B

UPS C PDU
D

A+B A + B-F Data Center

PUE = ERE = Boundary
D D
PUE is still the right metric; note that “A” (and PUE) will be less than in the design
w/o added recovery and absorption chiller
Copyright © 2011, The Green Grid
Comparison with PUE
• One view of PUE is that is the “tax” or burden in energy costs you
must pay above the IT load to run the data center; ERE allows the
same vision
• PUE = 1.0 means 100% of the energy you bring in to the data center
goes to the IT
• ERE = 1.0 means you only need to bring into the site an amount
equal to 100% of the IT energy to support the data center
• We need both!
• Case 1 Case 2
• PUE = 2.0 PUE = 1.2
• ERF= 0.55 ERF=0.25
• ERE = 0.9 ERE=0.9
• Case 1 focuses on PUE, Case 2 focuses on ERF

Copyright © 2011, The Green Grid

Data Center Compute
Efficiency (DCcE)
Unused Servers
• ~15% of servers powered on but not being used
• CPU utilization doesn‟t tell the whole story
• Virtualization doesn‟t cure the problem – it can
make it worse
• Removing unused servers will increase data
center efficiency
 Reduce power for physical servers
 Reduce resource usage for virtual servers

Copyright © 2011, The Green Grid

How to Find Unused Servers
• Servers are procured to provide a primary service
• If they are no longer providing this service then they are
no longer needed and should be decommissioned or
repurposed.
• Useful work <> CPU utilization
• Secondary & tertiary services can cause utilization
 Virus scanning
 Disk indexing / defragmentation
 Backup
 Etc..

Copyright © 2011, The Green Grid

Determining Primary Service Usage

• Tracking primary services resource utilization is

impractical – too many different primary service apps
• Tracking secondary & tertiary services is easier –
typically a well known small set of applications
• Primary service work = All work – secondary & tertiary
work
 True for CPU & I/O
• Some applications do not lend themselves to
measurement like this – e.g. terminal services
 Track incoming network sessions for „useful‟ processes
 Track interactive logons – always assumed to be useful

Copyright © 2011, The Green Grid

Calculating Primary Service Usage

• Over a time period

 If
– All CPU minus secondary & tertiary CPU > noise threshold
 Or
– All I/O minus secondary & tertiary I/O > noise threshold
 Or
– There have been incoming network sessions for primary services
 Or
– There has been an interactive logon
 Then
– Server was being useful
 Else
– Server was not being useful

Copyright © 2011, The Green Grid

Server Compute Efficiency (ScE)

• The proportion of samples that the server is

providing primary services over time (as a
percentage)
• Any server with an ScE of 0% over a prolonged
period is not being used and can be
decommissioned or repurposed
• Any servers with low ScE are worth investigation
– may be candidates for virtualization

Copyright © 2011, The Green Grid

Data Center compute Efficiency (DCcE)

• Data Center compute Efficiency (DCcE) aggregates ScE

across all servers in the data center
• Provides a benchmark against which to improve (like
PUE)
• DCcE is NOT a productivity metric – it does not measure
how MUCH work is done, just the proportion of work that
is useful
• DCcE CANNOT and SHOULD NOT be compared
between data centers due to the subjective
determination of secondary and tertiary processes

Copyright © 2011, The Green Grid

 Thank you for attending
The Green Grid Technical Forum 2011
Join The Green Grid Technical Forum
conversation on Twitter!
As you are tweeting, use #TGG11.