Data Centers

The following article was published in ASHRAE Journal, April 2005. Copyright 2005 American Society of Heating, Refrigerating and AirConditioning Engineers, Inc. It is presented for educational purposes only. This article may not be copied and/or distributed electronically or in paper form without permission of ASHRAE.

Meeting Data Center Temperature Requirements

By Roger Schmidt, Ph.D., Member ASHRAE; Madhusudan Iyengar, Ph.D., Member ASHRAE; and Richard Chu

he increasing compactness of data processing equipment size and the expanding needs of the information technology industry has led to more equipment being packed into the same amount of space. However, the energy efficiency of computing and storage equipment has not risen at the same rate. This has resulted in a signicant increase in power density and heat dissipation. Therefore, the cooling of computer and telecommunications equipment rooms is becoming a major challenge. The problem in many data centers is that the chilled air from the air-conditioning units cannot satisfy the local airow requirements of the computer and storage racks. This permits only a portion of each electronic equipment rack to be cooled by the chilled air, while other portions of the rack pull in air from other regions of the data center that is generally at a higher air temperature. The primary questions addressed here are: How much chilled air is supplied
44 ASHRAE Journal

to the rack compared to the airow rate through the rack? and Is the remainder of the air supplied to the rack within acceptable temperature limits? Data centers, such as the one in Figure 1, typically are arranged into hot and cold aisles. This arrangement accommodates most rack designs that use front-to-back cooling. And, it somewhat separates cold air exiting the perforated tiles (for raised oor designs) and overhead chilled airow (for non-raised oor designs) from the hot air exhausting from the rear of the racks. The racks are positioned so the fronts of the racks face the cold aisle as in Figure 1. Similarly, the rear of the racks face each other, and provide a hot air exhaust region. This layout allows the chilled air to wash the fronts of the data processing (DP) equipment while the hot air from the racks exit into the hot aisle as it returns to the inlet of the air-conditioning units. With the arrangement of DP equipment in rows within a data center, there may
ashrae.org

be zones where all equipment within that zone dissipates very high heat loads. This arrangement of equipment may be required to achieve the performance desired (cabling between computer racks may be restricted in length to achieve performance requirements). These high performance zones can provide significant challenges in maintaining an environment within the manufacturers temperature specications. Airflow distribution within a data center has a major impact on the thermal environment of the DP equipment located within these rooms. A key requirement of manufacturers is that the inlet temperature and humidity to the electronic
About the Authors Roger Schmidt, Ph.D., is distinguished engineer and chief thermal architect, and Madhusudan Iyengar, Ph.D., is an advisory engineer at the Systems & Technology Group at IBM in Poughkeepsie, N.Y. Richard Chu is IBM fellow and manager of the Advanced Thermal Labs at the Systems & Technology Group at IBM in Poughkeepsie, N.Y.

April 2005

The problem in many data centers is that the chilled air from the air-conditioning units cannot satisfy the local airow requirements of the computer and storage racks.
equipment be maintained within their specications. For a Class 1 datacom environment as specied in Reference 1, the allowable inlet air temperature is between 15C and 32C (59F and 89.6F) while the relative humidity is between 20% and 80%. Higher elevations require a de-rating of the maximum dry-bulb temperature of 1C (1.8F) for every 300 m (984 ft) above an elevation of 900 m (2952 ft) up to a maximum elevation of 3050 m (10,004 ft). These temperature/humidity requirements are to be maintained over the entire air inlet area of the rack. Three other class environments specied in Reference 1 generally have a wider range of environmental requirements. With airow through an electronic rack nominally front-to-back, the chilled air supply, whether from a raised oor tile or via diffusers from the ceiling, is typically only a fraction of the rack airow rate. This is due to the limitations of tile or diffuser ow rate. The remaining supply of air to a rack is made up of ambient room air from mixed hot and cold airstreams and/or recirculation of hot air exhausting from the rear of a rack then returning directly to its intake. This may be acceptable if the rack inlet air temperature is within the air temperature specications of the rack. However, the airow paths could result in a wide range in inlet air temperatures across the face of the rack. For a raised oor layout, the inlet air temperature could range from 10C to 15C (50F to 59F) at the bottom of the rack close to the chilled air supply to as much as 30C to 40C (86F to 104F) at the top of the rack where the warmer air
April 2005

is pulled from other regions of the data center to satisfy the ow required by the rack. If the temperature specications of the rack encompass this range, this would be acceptable, even though prolonged periods of operation at elevated temperatures may be detrimental to the reliability of the system. Reference 2 compared four ventilation schemes for cooling electronic equipment in a data center: underoor supply/overhead return, underoor supply/horizontal return, overhead supply/underoor return and overhead supply/horizontal return. Experimental results were presented, as well as an analytical model of the underfloor supply/overhead return showing the airow and temperature distribution at the inlet of the electronic equipment rack. The analytical model showed the rack intake air temperature increasing with height from the raised oor chilled air supply to the top of a rack where increased recirculation of exhaust air from the rear of a rack occurs. References 3 and 4 expand on some of the data shown in Reference 2.
Thermal Performance Metrics

chilled air entering the data center room from the air-conditioning units. All Track inlet temperature measurements reported here were made at a height of 1.75 m (5.74 ft) measured from the raised oor and at the air inlet of the 2 m (6.56 ft) high racks. The second temperature metric used here to thermally characterize the data center is the temperature difference ratio: RdT = dTinlet / dTrack (2)

where dTinlet is defined in Equation 1, and dTrack is the air temperature rise through the computer rack equipment. When this ratio, RdT , is 0, then the air entering the rack is at the supply chilled air temperature, and there is no recirculation (at the point of measurement). When this ratio is equal to 1, all of the air (at the point of measurement) is at the racks hot exhaust air temperature due to recirculation. When this ratio is greater than 1, it shows the presence of a self-heating loop (at the point of measurement), where the air is recirculating and increasing in temperature more than what would have been from one pass through the rack.
Data Center Measurements

While the absolute value of the air temperature that enters the rack is of ultimate importance, for design purposes and system characterization, two other rack inlet air temperature-related metrics have been used here. The rst metric is the temperature difference between the inlet air temperature (at a specic location in front of the rack) and the supply chilled air, which is given by: dTinlet = Track inlet Tchilled supply air (1)

where Track inlet is the air inlet temperature to the rack, and Tchilled supply air is the inlet

Measurements were taken in four distinctly different data centers with the results from only one reported here. Similar results were obtained from all four data centers. The results of this investigation are summarized in Figure 2, which refers to the layout depicted in Figure 3. All four data centers were thermally proled, that is, measurements made of temperature, power and airow. The measurement methodology and the results are described in detail for one of the data centers in Reference 5.
ASHRAE Journal 45

For the results reported here, successively larger areas of the 0.49. One interpretation would be that 49% of the ow for data center were examined for ow and temperature to see if the rack is supplied by the perforated tile in front of the rack trends might emerge that describe the mixing of the air and the while the remaining rack ow is drawn from elsewhere in resulting dTinlet at the air intake of the racks. the room. Although in most cases this would probably be the Region 1 shown in Figure 3 is part of a larger data center case, it cannot be ensured that all the chilled air ow from that encompassing 687 m2 (7,391 ft2). Four areas of the data center tile passes to the adjacent rack. However, forming this ratio were examined. The smallest area for the data center included for larger areas of the data center, a more accurate picture is one rack (Rack 1 in Figure 3) and the perforated tile area directly obtained. This ratio formed with the measurements from all in front of the rack. The rack chosen was the highest powered four data centers (only one is shown here for brevity) is quite rack of the two within the local area. clear; the ratios for all areas of all data centers examined fell The remaining areas examined were successively larger: Lo- between 0.28 and 0.68. cal Area 1 encompassing two racks facing one another (depicted And, for the data center measurements reported in Figure 2, by the area enclosed by the dashed lines in Figure 3), multiple the ratio for the rack, local, region and data center areas ranged racks facing one another and sharing the cold aisle (Region 1 from 0.48 to 0.63. This ratio of less than one but greater than 0 in Figure 3) and nally the entire data center (not shown). Each indicates that some air is supplied to the rack(s) from elsewhere in grouping of data in Figure 2 displays the results for each suc- the room and not from the perforated tiles in front of the racks. cessively larger area: rack, local, region and data center. The fth grouping in Figure 2 depicts the ratio of the meaThe first grouping shows sured cold aisle plus hot aisle Racks Air-Conditioning Unit power for the rack at 7.5 kW ow rates and the rack ow (25,600 Btu/h), local area rate. The cable openings are a power of 14.2 kW (48,460 source of chilled air from the Btu/h), regional power of 106 hot aisle and provide cooling kW (361,770 Btu/h) and the to the hot air exhausting from data center power of 1089 the rear of the racks. This ratio kW (3,716,720 Btu/h). (The varied from 0.73 to 1.0. last bar in each grouping is Even when including the from the numerical modeling, cable opening ow, the ratio which is described later.) was less than 1 for all cases The next grouping shows except for the region showing the heat flux for each area that in most cases the rack inlet with the highest occurring air is from other regions of the for the rack. The corresponddata center. In further support ing dTinlets are shown in the of warm air recirculation the Perforated Floor Tiles next grouping (averages are RdT, shown in the last groupshown when displaying val- Figure 1: A data center arranged in hot and cold aisles. ing, fell between 0.5 and 0.75 ues for more than one rack). indicating that a portion of the With a dT inlet of 8.1C (14.6F) for Rack 1 as in Figure 2 and air for the rack was supplied from elsewhere in the room. a supply chilled air temperature of 12.9C (55.2F), the rack Reference 5 states that based on the reported measurements in inlet air temperature (at a height of 1.75 m [5.74 ft]) of 21C that study if the perforated tiles associated with a rack exhausted (69.8F) is well below the rack maximum temperature limit of one-quarter to one-half of the rack ow, then the system inlet air 32C (89.6F). temperature could be maintained (given the chilled air exhaust Although not shown, all rack inlet temperatures in the data temperature was approximately 13C to 15C [55.4F to 59F]). center were within the maximum rack inlet temperature speci- For this study, and in all cases of the data center (Rack 1, Local cation. And, every rack inlet air temperature was greater than Area 1, Region 1 and Data Center 1) , the ratio of chilled airow the supply chilled air temperature showing that warm air from exhausting from the perforated tiles to the rack ow rate varied other portions of the room is being drawn into the rack intakes. between 0.48 and 0.63 (the range for all four data centers was Another way to conrm that warm air is entering the racks is 0.28 to 0.68) , and the air inlet temperature for the racks were to examine the airow rates. within specications. This study shows that the data is similar The fourth grouping of Figure 2 displays the perforated tile to that reported in Reference 5. ow rate immediately in front of the rack(s) compared to the rack ow rate(s). For example, the ow rate for Rack 1 is 0.5 Modeling Experiments m3/s (1,050 cfm) while the perforated tile ow rate immediately One of the objectives of the modeling was to construct a in front of the rack is 0.24 m3/s (516 cfm) forming a ratio of simplied model of the region (described previously) used
46 ASHRAE Journal ashrae.org April 2005

2.25 2 1.75 1.5 Measured Value 1.25 1 0.75 0.5 0.25 0 10 10 Rack Local Region Data Center Numerical Model 10 102 103 103 Heat Flux kW/m2 10 Power kW dT Inlet, C

Perf. Flow/ Rack Flow

Perf. + Cable Flow/ Rack Flow

RdT

Figure 2: Results for measurement and modeling of a data center shown below in Figure 3.

for measurements but still capture most of the thermo-uid The rack was modeled with blowers that force a specic and geometric parameters and then compare the results. A airow through these three portions of the rack with the same representative model of a section of raised oor data center ow characteristics as the rack. The temperature of the chilled system, depicted in Figure 4, was constructed using a com- air entering the room through the perforated tiles was xed mercial software tool. The model was constructed to closely at 15C (59F). The rack ow rate, perforated tile ow rates, resemble the layout in Figure 3, for and cable ow rates matched those Region 1 which measurements were taken. measured in Region 1 of Figure 3. The model consists of four priThe room dimensions were four mary components; namely, the heat times that of the quarter symmetry generating datacom equipment model shown in Figure 4, i.e., 6.05 racks, perforated tiles that supply m by 2.8 m (19.8 ft by 9.18 ft), with chilled air, cable openings that a room height of 2.75 m (9.02 ft). cause underfloor air to leak into The rack dimensions were 0.61 m the hot aisle, and computer room (2 ft) wide by 1.22 m (4 ft) deep by air-conditioning units (CRAC) 2 m (6.56 ft) tall. that draw in hot return air from the The resulting heat flux of this racks and exhaust chilled air into conguration is 1.6 kW/m2 (507.6 the raised oor plenum. The model Btu/h/ft2) (Figure 2). Thus, the heat Rack 1 comprised of a quarter symmetry ux and the ow conditions are the Local Area 1 of 16 rack sections of a data center same for the numerical model cell arranged in a cold aisle-hot aisle Figure 3: Rack layout in a data center. as for those measured in the data fashion. The racks were assumed to center and reported previously. One dissipate the same heat load as the machines for which the perforated tile with dimensions of 0.61 m by 0.61 m (2 ft by experimental results are reported. The geometry of some of 2 ft) was placed in front of each rack. The cable openings at the racks from the eld measurements was replicated in the the back of the rack were modeled as 17.8 cm (7 in.) deep computer model. and the width of the rack. The racks were modeled to dissipate 6.7 kW (22,870 Btu/h) The thermo-uid interactions were numerically solved using each (average of the racks measured in the data center), with the k-e turbulence model, which is the preferred choice for such the heat load distributed in three portions of the rack (same as data center uid ow modeling. The cell count for the models those shown in the measurements of Figure 3): input/output was approximately 50,000. Temperature data was recorded for portion at the bottom of the rack, central electronic complex the numerical simulations at nine vertical locations in front on that includes memory and processors in the center of the rack, the rack, namely at heights of 200 to 1600 mm (7.87 to 63 in.) and ac/dc power supplies located at the top of the rack. in increments of 200 mm (7.87 in.), and the highest point at
April 2005 ASHRAE Journal 47

1750 mm (68.9 in.) (same as the location for measurements). The criteria for satisfactory convergence took into consideration the trends and absolute values, respectively, for the temperature, continuity residuals and the monitor point temperatures.

2. Although the room ow rate is adequate to cool the overall heat load in the data center and the local ow rate adjacent to the racks does not appear adequate, the convection currents that occur at the room level are adequate to bring the local air temperatures for the high powered racks within the rack air temperature specications. Flow patterns observed from Measurements and Modeling Comparisons Although several comparisons can be made between the simple modeling of a data center show the upper regions of a measurements reported from Data Center 1 and the numeri- rack being satised with ows from other portions of the room. cal modeling, the desire is to nd a metric that can easily be Mixing of the cold and hot airstreams can occur in the vicinity used in data center characterization and, therefore, the focus of the rack or in other portions of the data center. However, the was on the dimensionless metric RdT dened in Equation 2. mixing that can occur allows the air temperature to be within As seen in Figure 2, the temperature data from the numerical the air temperature specications of the racks. 3. A novel methodology was demonstrated by which a large results were higher than those measured, for the data center complex data center can be level case, when the airow Chilled Air-Conditioning Unit separated into global, region, ratios and the heat ux were local, and rack level regions modeled to be same as the for analysis and modeling. experimental case. This dem4. A temperature difference onstrates the need for signiformed by the temperature cantly more work in this area rise of the rack inlet air and if these tools are to be used the supply chilled air temperamore generally to model the ture, and a temperature ratio thermal and airow behavior formed by the temperature of data centers. rise of the rack inlet air and the Several weaknesses need to Air Temp., C air temperature rise through be further explored based on > 48 the rack, was used to colthe following observations: 39.75 lapse the data for comparison The difficulty in making 31.5 Cable Openings purposes. detailed and exhaustive 23.25 Racks Perforated 5. A comparison between measurements at the data Symmetry Plane Tiles <15 measurement data and model center level due to the size Figure 4: Computer-based simulation of a data center. results showed the need for of the facility; more work in this area. The complex nature of the rack design, which does not always lend itself to simplicaAcknowledgments tion in numerical modeling; and We want to thank the people who aided in the collection of the The large number of variables and boundary conditions that data for this study: Bob Wasilewski and Tom Juliano of DLB Asneed to be matched to make an accurate comparison. Future work will be devoted to addressing these issues to sociates Consulting Engineers, Ocean, N.J., and Saurabh Shrivatsava of the State University of New York at Binghamton. close the gap between modeling and measurements.
Summary/Conclusions References
1. Thermal Guidelines for Data Processing Equipment. 2004. ASHRAE Special Publication. 2. Nakao, M., H. Hayama, M. Nishioka. 1991. Which cooling air supply system is better for a high heat density room: underoor or overhead? Thirteenth International Telecommunications Energy Conference, (INTELEC 91), Paper 12-4, pp. 393 400. 3. Hayama, H. M. Nakao. 1989. Air ow systems for telecommunications equipment rooms. The Eleventh International Telecommunications Energy Conference (INTELEC 89), Vol. 1, Paper 8.3, pp. 17. 4. Hayama, H., M. Nakao, and M. Sanabe. 1990. Airow distribution in telecommunications equipment rooms. Twelfth International Telecommunications Energy Conference (INTELEC 90), Paper 11-4, pp. 206 212. 5. Schmidt, R. 2004. Thermal prole of a high density data centermethodology to thermally characterize a data center. 2004 ASHRAE Annual Meeting. April 2005

1. Measurements in a data center showed with signicantly less ow from the perforated tiles in front of racks, compared to the rack ow, the inlet air temperatures into a rack could be met. Specically, measurements showed that for the rack, local area, region and data center the ratio of perforated tile ow to rack ow ranging from 0.48 to 0.63 were acceptable to meet inlet air temperature. (The data from four data centers ranged from 0.28 to 0.68). This range was similar to that reported in Reference 5, stating that if the perforated tiles associated with a rack exhausted one-quarter to one-half the ow of the rack, then the system inlet air temperature could be maintained (given the chilled air exhaust temperature was below approximately 15C [59F]).
48 ASHRAE Journal

ashrae.org