ASHRAE Thermal Guidelines

1
 Agenda
•  ASHRAE and TC9.9
•  Thermal Guidelines (Third Edition)
•  Air-cooling
•  Liquid-cooling
•  Air-cooling thermal guidelines in practice
•  Performance
•  Reliability

2
 ASHRAE and TC 9.9
• ASHRAE
•  American Society of Heating, Refrigeration and Air-
Conditioning Engineers

• Technical Committee 9.9

•  Technical Committee 9.9 – Mission Critical Facilities, Data
Centers, Technology Spaces, & Electronic Equipment
(http://tc99.ashraetcs.org/index.html)
•  Environment – thermal, structural, design, testing of data
centers and technology spaces for IT equipment

• Committee Members
•  IT equipment manufacturers
•  Environmental equipment manufacturers
•  Data center designers
•  End users

3
 TC 9.9 Datacom Book Series

1.  Thermal Guidelines for Data Processing Environments 3rd Edition

2.  Datacom Equipment Power Trends & Cooling Applications 2nd Edition
(2012)
3.  Design Considerations for Datacom Equipment Centers (2006)
4.  Liquid Cooling Guidelines for Datacom Equipment Centers 2nd Edition
(2014)
5.  Structural & Vibration Guidelines for Datacom Equipment Centers
(2008)
6.  Best Practices for Datacom Facility Energy Efficiency (2008)
7.  High Density Data Centers – Case Studies & Best Practices (2008)
8.  Particulate & Gaseous Contamination in Datacom Environments (2009)
9.  Real-Time Energy Consumption Measurements in Data Centers (2009)
10. Green Tips for Data Centers (2011)
4
 Thermal Guidelines for Data Processing
Environments
Chapter 1 – Introduction
Chapter 2 – Environmental Guidelines
for Air-Cooled Equipment
Chapter 3 – Environmental Guidelines
for Liquid-Cooled Equipment
Chapter 4 – Facility Temperature & Humidity
Measurement
Chapter 5 – Equipment Placement & Airflow
Patterns
Chapter 6 – Equipment Manufacturers Heat
& Airflow Reporting

5
 Air-Cooled IT Equipment Class Ranges

•  Allowable Range – ITE designed to function within this

range
•  Recommended Range – Guidance from ITE
manufacturers for high reliability, minimal power
consumption (of ITE) and maximum performance
•  Operating Range – Actual limits for an individual
datacenter
•  Can begin with the recommended range
•  Dependent upon datacenter operator’s evaluation of
ITE factors described in the whitepaper in
combination with knowledge of the datacenter
design

6
 2011 ASHRAE Thermal Guidelines

Recommended  

7
 Operating Envelope

ITE ITE ITE

Allowable Corrosion Acoustical
Envelope Noise

ITE ITE Cost

Performance

ITE
ITE Power Airflow

Operating Datacenter
ITE Envelope operational
Reliability knowledge

8
 Liquid-Cooling Environments

Reference
Point

Facility IT Equipment

Atmosphere ← Building ← Room ← Aisle ← Rack ← System ← Board ← Chip

9
 2011 ASHRAE Liquid-Cooled Guidelines
Supplemental Facility
Main Cooling
Classes   Cooling Supply Water
Equipment  
Equipment   Temp (°C)  
Water-side
W1   Economizer 2 – 17  
Chiller/Cooling Tower  
(w/ drycooler or
W2   cooling tower)   2 – 27  
W3   Cooling Tower   Chiller   2 – 32  
Water-side Economizer
W4   (w/ drycooler or N/A   2 – 45  
cooling tower)  
W5   Building heating system Cooling tower > 45  

10
 ITE Thermal Design
Component Temperature

Damage
Shutdown
Limit

Performance
Functional and Power
Limit Management

Reliability Manage
Limit Cooling
Optimize Performance, Power, Acoustics

11
 CPU Performance – Light Load
21°C 35°C 45°C
PO Turbo

A4 Workload-tuned
A4 Workload-tuned
A4 Workload-tuned
P1 Design
Lower Frequency

P2
Power Saving,

• 
A4 server

A2 server
A2 server

A4 server
A2 server

A4 server
• 
• 
Pn

T Damage Protection

Unpredictable performance, memory follows a similar trend

although throttling mechanism is different
12
 CPU Performance – Heavy Load
21°C 35°C 45°C
PO Turbo

A4 Workload-tuned
A4 Workload-tuned

A4 Workload-tuned
P1 Design
Lower Frequency

P2
Power Saving,

• 
A4 server

A2 server

A4 server
A2 server

A4 server
A2 server
• 
• 
Pn

T Damage Protection

Performance – dependent on adequacy of thermal design

13
 Reliability ‘X-Factor’

Assumes  con*nuous  (7  x  24  x  365)  opera*on  with  Dry  Bulb  Temp.  at  ITE  inlet.    

14
 Server Reliability Trend vs. Ambient Temperature
(cont.)
Ÿ  2010 dry bulb temp for Chicago with air mixing to maintain 59°F min. temp.

15
 Application of X-Factor (Chicago example)
Ÿ  Cross referencing the hardware failure rates with the dry bulb data for
Chicago, we can establish the Net X-Factor across the whole year.

T  he  Net  X-­‐Factor  for  in  Chicago  IT  hardware  failure  with  compressor-­‐less  
cooling  and  a  variable  data  center  temperature  is  almost  exactly  the  SAME  
as  if  the  data  center  was  operaDng  at  a  Dghtly  controlled  temp.  of  68°F.  

16
 City Comparison: Reliability
Ÿ  Average Net Failure Rate Comparison for Air-side
Economization (US Cities)

17
 City Comparison: Chiller Hours
Ÿ  Chiller Hours Per Year for Air-side Economization (US Cities)

18
 Backup

19
 ETSI

20
 ASHRAE Environmental Classes
2011 2008 Applications IT Equipment Environmental Control
Enterprise servers, storage
A1 1 Tightly controlled
products
Volume servers, storage products,
A2 2 Some control
personal computers, workstations
Datacenter
Volume servers, storage products,
A3 NA Some control
personal computers, workstations

Volume servers, storage products,

A4 NA Some control
personal computers, workstations

Office, home, Personal computers,

B 3 transportable workstations, laptops, and Minimal control
environment, etc. printers

Point-of-sale, Point-of-sale equipment,

C 4 industrial, ruggedized controllers, or No control
factory, etc. computers and PDAs

21
 2011 Thermal Guidelines – Power and Airflow
–  Power – ITE thermal management drives air movers to cool components
–  Air movers, some silicon devices consume more power with elevated temperature
–  Airflow required not linear with temperature

1.25
Server  Power   2.81.25
Server  Airflow  
2.6

1.20 1.20

- xIncrease
2.4

factor
Server Power Increase - x factor

2.2
1.15 1.15

Flowrate
2.0

Air Increase
1.8
1.10 1.10

Power
1.6

1.05 Server
Server 1.41.05

1.2

1.00 1.01.00
10 15 20 25 30 35 40 10 10 1515 20
20 25 25 30 30 35 35 40
Ambient Temperature - C AmbientAmbient Temperature
Temperature -C
- degrees C

22