HVAC System Design

Mark Hydeman, P.E., FASHRAE

Taylor Engineering, LLC
mhydeman@taylor-engineering.com
Taylor Engineering, LLC 1
How do you effectively fight a fire?
it takes 2,000 to 3,000 times the
volume of air to cool what you can
with water!
With air, or… with water?

Taylor Engineering, LLC 2

State of the present: with air

Taylor Engineering, LLC 3

Air system design overview
 Data center layout
 Airflow configurations
 Distribution:
overhead or underfloor
 Control: constant or variable volume
 Airflow issues
 Economizers
 Humidity control issues

Taylor Engineering, LLC 4

Data center layout

Server airflow front to

back or front to back and
top are recommended

Cold Aisle

Hot Aisle

© 2004, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. (www.ashrae.org). Reprinted by permission from
ASHRAE Thermal Guidelines for Data Processing Environments. This material may not be copied nor distributed in either paper or digital form
without ASHRAE’s permission.

Taylor Engineering, LLC 5

 Data center layout

Underfloor Supply

Cold Aisle

Hot Aisle

Only 1 pressure
zone for UF!
© 2004, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. (www.ashrae.org). Reprinted by permission from
ASHRAE Thermal Guidelines for Data Processing Environments. This material may not be copied nor distributed in either paper or digital form
without ASHRAE’s permission.

Taylor Engineering, LLC 6

 Data center layout
You can
incorporate VAV
on each branch

Overhead Supply

Cold Aisle

Hot Aisle

© 2004, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. (www.ashrae.org). Reprinted by permission from
ASHRAE Thermal Guidelines for Data Processing Environments. This material may not be copied nor distributed in either paper or digital form
without ASHRAE’s permission.

Taylor Engineering, LLC 7

Typical temperature profile with UF supply
Too hot Too hot

Just right

Too cold

Elevation at a cold aisle looking at racks

There are numerous references in ASHRAE. See for example V. Sorell et al; “Comparison of
Overhead and Underfloor Air Delivery Systems in a Data Center Environment Using CFD
Modeling”; ASHRAE Symposium Paper DE-05-11-5; 2005

Taylor Engineering, LLC 8

Typical temperature profile with OH supply
Too warm Too warm

Just right

Elevation at a cold aisle looking at racks

Taylor Engineering, LLC 9

 Hot aisle lid

Aisle capping
End cap
Cold Aisle Caps
© APC reprinted with permission

Cold Aisle

Hot Aisle

© 2004, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. (www.ashrae.org). Reprinted by permission from
ASHRAE Thermal Guidelines for Data Processing Environments. This material may not be copied nor distributed in either paper or digital form
without ASHRAE’s permission.

Taylor Engineering, LLC 10

Aisle capping
LBNL has recently performed
research on aisle capping
Cold Aisle Caps

Cold Aisle

Hot Aisle

© 2004, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. (www.ashrae.org). Reprinted by permission from
ASHRAE Thermal Guidelines for Data Processing Environments. This material may not be copied nor distributed in either paper or digital form
without ASHRAE’s permission.

Taylor Engineering, LLC 11

 Overhead (OH) vs. Underfloor (UF)
Issue Overhead (OH) Supply Underfloor (UF) Supply
Capacity Limited by space and aisle velocity. Limited by free area of floor tiles.
Balancing Continuous on both outlet and branch. Usually limited to incremental changes by
diffuser type. Some tiles have balancing
dampers. Also underfloor velocities can
starve floor grilles!
Control Up to one pressure zone by branch. Only one pressure zone per floor, can
provide multiple temperature zones.
Temperature Most uniform. Commonly cold at bottom and hot at top.
Control
First Cost Best (if you eliminate the floor). Generally worse.
Energy Cost Best. Worst.
Aisle Capping Hot or cold aisle possible. Hot or cold aisle possible.

Taylor Engineering, LLC 12

Airflow design disjoint
 IT departments select servers and racks
 Engineers size the fans and cooling
capacity
 What’s missing
in this picture?

Taylor Engineering, LLC 13

Airflow with constant volume systems

 Hot spots
 Higher hot aisle
temperature
 Possible equipment
failure or degradation

 
V HVAC _ Supply   V Servers
Taylor Engineering, LLC 14
Airflow with constant volume systems

 Least hot spots

 Higher air velocities
 Higher fan energy
 Reduced economizer
effectiveness (due to
lower return
temperatures)
 

Taylor Engineering, LLC

V HVAC _ Supply   V Servers 15
Airflow with constant volume systems

 Note most of these observations apply to

overhead and underfloor distribution
 With constant volume fans on the servers
you can only be right at one condition of
server loading!
 The solution is to employ variable speed
server and distribution fans…

Taylor Engineering, LLC 16

Airflow with variable volume systems

Partial flow condition

 Best energy performance
but difficult to control

 
V HVAC _ Supply   V Servers
Taylor Engineering, LLC 17
How Do You Balance Airflow?

 Spreadsheet
 CFD
 Monitoring/Site
Measurements

Image from TileFlow

http://www.inres.com/Products/TileFlow/tileflow.html,
Used with permission from Innovative Research, Inc.

Taylor Engineering, LLC 18

Thermal report

From ASHRAE’s Thermal Guidelines for Data Processing Environments

Taylor Engineering, LLC 19
What’s the server airflow?
SUN SUN DELL DELL
V490 V240 2850 6850
num fans 9 3 n/a n/a
total CFM (max) 150 55.65 42 185
total CFM (min) 27 126
fan speed single speed variable 2 speed 2 speed
fan control n/a inlet temp. 77F inlet 77F inlet
Form Factor (in U's) 5 2 2 4
heat min config (btuh) 798 454
heat max config (btuh) 5,459 1,639 2,222 4,236
heat max (watts) 1,599 480 651 1,241
dT min config - 13 - 3
dT max config 33 27 48 21
servers per rack 8 21 21 10
CFM/rack (hi inlet temp) 1,200 1,169 882 1,850
CFM/rack (low inlet temp) 1,200 567 1,260
max load / rack (kW) 13 10 14 12

Taylor Engineering, LLC 20

Best air delivery practices
 Arrange racks in hot aisle/cold aisle configuration
 Try to match or exceed server airflow by aisle
 Get thermal report data from IT if possible
 Plan for worst case
 Get variable speed or two speed fans on servers if possible
 Provide variable airflow fans for AC unit supply
 Also consider using air handlers rather than CRACs for improved
performance (to be elaborated on later)
 Use overhead supply where possible
 Provide aisle capping (preferably cold aisles, refer to LBNL
presentation for more details)
 Plug floor leaks and provide blank off plates in racks
 Draw return from as high as possible
 Use CFD to inform design and operation

Taylor Engineering, LLC 21

Air-side economizer

Taylor Engineering, LLC 22

Air-Side Economizer issues
 Hygroscopic dust
 LBNL is doing some research on this
 Design humidity conditions
 See following slides

Taylor Engineering, LLC 23

Design conditions at the zone

© 2005, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. (www.ashrae.org). Reprinted by permission from
ASHRAE Design Considerations for Data and Communications Equipment Centers. This material may not be copied nor distributed in either
paper or digital form without ASHRAE’s permission.

Taylor Engineering, LLC 24

 San Francisco
40 .020

80
75 .019

.018

.017

PSYCHROMETRIC 35

75
CHART 70 WE
TB
UL
BT
.016

Normal Temperature EM
PE
RA
TU
.015

R
I-P Units E-
°F .014
30
16 FEET 65

HUMIDITY RATIO - POUNDS MOISTURE PER POUND DRY AIR

70
Upper Allowed Humidity Limit .013
BAROMETRIC PRESSURE: 29.904 in. HG
.012

14.0
San Francisco Climate Data Bins 60

VOL
65 .011
with Data Center Guideline Zones 25

UME
.010

- CU
Weather Hours Design Target

.FT.
55
360 to 321 60 .009

PER
320 to 281
280 to 241

LB.
240 to 201 90
%
X % .008

D
20 50 25

RY
200 to 161
160 to 121 55

A
%
80

IR
120 to 81 .007
Class1;Recommend
80 to 41 45
40 to 1 %
70
50 .006

Negligible time of possible 15 40 60

%
Class1;Allow
.005
45
concern for humidification 35 50%
15%

.004
40 NEBS;Recommend
13.0

40%

10 35 .003
30%
30
.002
20%
Lower Allowed Humidity Limit (20%RH)
E HUMIDITY .001
10% RELATIV

30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

Chart by: HANDS DOWN SOFTWARE, www.handsdownsoftware.com DRY BULB TEMPERATURE - °F

Taylor Engineering, LLC 25

 Los Angeles
40
.020
80
75 .019

.018

.017

PSYCHROMETRIC 35

75

CHART 70 WE
TB
UL
BT
.016

EM
Normal Temperature PE
RA
TU
.015

RE
I-P Units -°
F .014
30
65
105 FEET 70

HUMIDITY RATIO - POUNDS MOISTURE PER POUND DRY AIR

Upper Allowed Humidity Limit .013
BAROMETRIC PRESSURE: 29.808 in. HG
.012

Los Angeles Climate Data Bins 60

14.0
65 .011
with Data Center Guideline Zones 25

VOL
UME
.010

Weather Hours Design Target

- CU
55
396 to 353 60 .009

.FT.
352 to 309

P
308 to 265
X

ER
%
264 to 221 20 90 % .008
25

LB.
50
220 to 177

D
176 to 133 % 55

RY
80 .007
132 to 89 Class1;Recommend

A
88 to 45

IR
45
%
44 to 1 70
50 .006

Only a few hours of possible 15 40 60

%
Class1;Allow
.005
45
concern for humidification 35 50%
15%

.004
40 NEBS;Recommend
40%
13.0

35 .003
10
30 30%

.002
20%
Lower Allowed Humidity Limit (20%RH)
E HUMIDITY .001
10% RELATIV

30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

Chart by: HANDS DOWN SOFTWARE, www.handsdownsoftware.com DRY BULB TEMPERATURE - °F

Taylor Engineering, LLC 26

 Sacramento
40 .020

80
75 .019

.018

.017

PSYCHROMETRIC 35

75
CHART 70 WE
T BU
LB
.016

TE
Normal Temperature MP
ER
AT
.015

UR
I-P Units E-
°F .014
30
26 FEET 65

HUMIDITY RATIO - POUNDS MOISTURE PER POUND DRY AIR

70
Upper Allowed Humidity Limit .013
BAROMETRIC PRESSURE: 29.893 in. HG
.012

14.0
Sacramento Climate Data Bins 60
65 .011

VOL
with Data Center Guideline Zones 25

UME
.010

- CU
Weather Hours 55
Design Target

.FT.
270 to 241 60 .009

PER
240 to 211
210 to 181
X

LB.
%
180 to 151 20 90 % .008
25

D
50
150 to 121

RY
120 to 91 % 55

A
80

IR
90 to 61 .007
Class1;Recommend
60 to 31 45
30 to 1 %
70
50 .006

Negligible time of possible 15 40 60

%
Class1;Allow
.005
concern for humidification 35
45
50%
15%

.004
40 NEBS;Recommend
13.0

40%

10 35 .003
30%
30
.002
20%
Lower Allowed Humidity Limit (20%RH)
E HUMIDITY .001
10% RELATIV

30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

Chart by: HANDS DOWN SOFTWARE, www.handsdownsoftware.com DRY BULB TEMPERATURE - °F

Taylor Engineering, LLC 27

Lower humidity limit
 Mitigate electrostatic discharge (ESD)
 Recommended procedures
 Personnel grounding
 Cable grounding
 Recommended equipment
 Grounding wrist straps on racks
 Grounded plate for cables
 Grounded flooring
 Servers rated for ESD resistance
 Industry practices
 Telecom industry has no lower limit
 The Electrostatic Discharge Association has removed humidity control as a primary
ESD control measure in their ESD/ANSI S20.20 standard
 Humidity controls are a point of failure and are hard to maintain
 Many data centers operate without humidification
 This needs more research
 And for some physical media (tape storage, printing and bursting)
 Old technology not found in most data centers
 It is best to segregate these items rather than humidify the entire data center
Taylor Engineering, LLC 28
ESD control: floor grounding

Image from Panduit, reprinted with permission

Taylor Engineering, LLC 29

Water-Side Economizer
Integrated

Heat
Exchanger in
series with
chillers on
CHW side
Economizer Summary
Air-Side Economizers
 Provides free cooling when
dry-bulb temperatures are
below 78°F-80°F.
 May increase particulates
(LBNL research indicates this
is of little concern).
 Should be integrated to be
most effective.
 Improves plant redundancy!
 Can work in conjunction with
water-side economizers on
data centers!
 Need to incorporate relief.
Both are proven technologies on data centers!
Taylor Engineering, LLC
Water-Side Economizers
 Provides low energy cooling
when wet-bulb temperatures
are below 55°F-60°F.
 Avoids increased particulates
(and low humidity if that
concerns you).
 Should be integrated to be
most effective (see previous
slide).
 Improves plant redundancy!
 Can work in conjunction with
air-side economizers on data
centers!
31
A case study of two designs
 Collocation facility in the
Bay Area
 Side by side designs in
same facility over two
phases
 Motivation for the second
design was to reduce
cost
Taylor Engineering, LLC
 Case study was
developed by Lawrence
Berkeley National
Laboratory (LBNL)
 Data Centers 8.1 and 8.2
 Both sections at ~30%
build-out during
monitoring
32
A tale of two designs: overview
 Phase 1 Data Center (8.1)
 26,200 ft2
 27 W/ft2 design
 Traditional under-floor design
with CRAC units
 Air-cooled DX
 Humidity controls (45%-55%)
Taylor Engineering, LLC
 Phase 2 Data Center (8.2)
 73,000 ft2
 50 W/ft2 design
 Under-floor supply from
central AHUs with CHW coils
 Water-cooled plant
 Air-side economizers
 No humidity controls
33
A tale of two designs: a closer look
kWcooling _ systems
Normalized efficiency metric: cooling  kWservers

1.60 Phase 1 Data Center (8.1) ~1/4 of the

1.40
Phase 2 Data Center (8.2) normalized energy
1.20
Normalized energy

1.00

0.80

0.60

0.40

0.20

-
Computer Loads UPS Losses HVAC Lighting

Data normalized to computer loads

Taylor Engineering, LLC 34
A tale of two designs: results
 Phase 1 Data Center (8.1)  Phase 2 Data Center (8.2)
 Around 2x the HVAC  Preferred by the facility
installed cost ($/ft2) operators and data center
personnel
 Around 4x the energy bills
(when normalized to server
load)
 Acoustical problems
 Higher maintenance costs
 Lost floor space in data
center due to CRACs

Taylor Engineering, LLC 35

Two data centers: summary
 What made the difference?
 Airsideeconomizers
 No humidity controls
 Water-cooled chilled water system
 AHUs instead of CRAC units

Taylor Engineering, LLC 36

Custom CRAH Unit (Large)

Taylor Engineering, LLC 37

Example CRAH Unit Comparison
Option 1 Option 2
Model Std CRAC Custom Model 1 Custom Model 2
Budget Cost $ 16,235 $ 23,000 $ 41,000
Number of units 21 13 4
net total cooling (btuh) 434,900 410,000 841,000
net sensible (btuh) 397,400 399,000 818,000
sensible (tons) 33.1 33.3 68.2
CFM 16,500 25,000 50,000
SAT 49.90 59.30 59.00
airside dT 25.10 15.70 16.00
Internal SP 2 0.8 0.8
1.8 1.8
no. fans 3 3 2
fan type Centrifugal Plenum Plenum
no. motors 1 3 2
HP/motor 15 5 15
total HP 15 15 30
BHP/motor 15 4.7 11.5
Unit BHP 15 14.1 23
unit width 122 122 122
depth 35 36 72
height 76 156 168
filter type ASHRAE 20% MERV 13 MERV 13
Water PD (ft) 13.5 ft 11.1 11.1
CHW dT 14F 20 20
GPM 66.80 44.00 88.00
Total GPM 1,403 924 66%
Total BHP 315 275 87%

Taylor Engineering, LLC 38

Example CRAH Unit Comparison

 34% less water flow

 13% less fan energy
 More if you consider the supply air temperature and airflow issues
 Excess fan capacity on new units
 36% higher cost for units, but
 Fewer piping connections
 Fewer electrical connections
 Fewer control panels
 No need for control gateway
 Can use the existing distribution piping and pumps (case study)
 Can use high quality sensors and place them where they make sense
 Possibly less turbulence at discharge?

Taylor Engineering, LLC 39

Air cooling issues
 Limitations on the data densities served (~200w/sf)
 Air delivery limitations
 Real estate
 Working conditions
 Hot aisles are approaching OSHA limits
 Costly infrastructure
 High energy costs
 Management over time
 Reliability
 Loss of power recovery
 Particulates

Taylor Engineering, LLC 40

Take Aways
 Use air- or water-side economizers where possible
 Consider personal grounding in lieu of humidification
 Consider AHUs as an alternative to CRACs
 Consider VSDs on fans, pumps, chillers and towers
 Refer to ASHRAE, LBNL and Uptime Institute for more
recommendations

Taylor Engineering, LLC 41

State of the future: cooling with liquid

Taylor Engineering, LLC 42