Applied Thermal Engineering 169 (2020) 114857

Contents lists available at ScienceDirect

Applied Thermal Engineering

journal homepage: www.elsevier.com/locate/apthermeng

Development and experimental study of an independent row-based cooling T

system for improving thermal performance of a data center
Jinkyun Choa,⁎, Jesang Woob
a
Department of Building and Plant Engineering, Hanbat National University, 34158 Daejeon, Republic of Korea
b
Infra Facility Engineering Team, SK Telecom Co., Ltd., Seoul 04539, Republic of Korea

HIGHLIGHTS

• Data center cooling with different air distribution strategies experimentally studied.
• We investigated how to remove IT heat effectively in data centers using full-scale field measurements.
• Two different cooling systems were evaluated based on six data center performance indices.
• New independent row-based cooling system provides high standards of thermal performance.
• Row-based cooling gives the less air distribution path, cooling capacity and fan power vs. room-based cooling.

ARTICLE INFO ABSTRACT

Keywords: As the power density of IT equipment increases, the requirements for data center facilities are also becoming
Data center more rigorous. In addition to higher IT power density, cooling has become a key issue in data centers where
Room-based cooling cooling accounts for about 40% of total energy usage. An improved independent row-based data center cooling
Row-based cooling system has been suggested, and is found to be the better efficient in removing heat output of IT equipment based
Thermal performance
on the observations and experimental results from two cooling strategies. This study, that is practice-based
Site measurement
Evaluation metrics
learning in a legacy data center remodeling project, has evaluated the energy and thermal performance of two
data center cooling approaches: an improved independent row-based cooling and a conventional room-based
cooling. The air temperature, humidity and air distribution efficiency of an IT environment were examined by
using field measurements and six performance indices. The differences and technical features of this experi-
mental study is that two different forms of cooling systems were evaluated based on the same IT environment
conditions; location of the IT equipment and the cooling load in same IT server room. The results show that data
center cooling systems utilizing the “the fully containment of cold and hot aisles” technique are very effective at
removing heat load of IT servers. The advantage of row-based cooling is that it is located close to hot spots and
does not experience the heat losses generated in the air delivery path of the room typical of room-based cooling
systems. SHI and RHI were improved by 37.1% and 20.0%, and RTI and β were improved by 73.2% and 44.3%
respectively. According to the observations and index evaluation results, we suggest that a new independent
row-based cooling strategy is more efficient for server cooling. Air management helps to reduce cooling energy
by enhancing optimal operation and improving cooling system efficiency. Furthermore, metrics play a sig-
nificant role in providing performance indices for air management systems.

1. Introduction
1.1. Background and objectives
The growth of the ICT industry, which includes artificial intelligence
(AI), the Internet of Things (IoT), and other leading areas of the 4th
Industrial Revolution, has been accompanied by continuous expansion
of the data center market. The energy use of data centers has doubled
every four years for the last decade. The increasing power usage and
removal of heat generated from IT equipment have been the main issues
of consumers [1]. PUE (Power Usage Effectiveness) [2] is an interna-
tional indicator used for evaluating the energy efficiency of a data
center and is measured by the ratio of the net power usage of IT
equipment to the total power usage, including that for cooling and

⁎
Corresponding author.
E-mail address: jinkyun.cho@hanbat.ac.kr (J. Cho).

https://doi.org/10.1016/j.applthermaleng.2019.114857
Received 30 March 2019; Received in revised form 17 November 2019; Accepted 26 December 2019
Available online 28 December 2019
1359-4311/ © 2019 Elsevier Ltd. All rights reserved.
J. Cho and J. Woo
Nomenclature
Q heat transfer rate [W]
T temperature [°C]
T temperature differential [°C]
m air flow rate [m3/h]
n total number of racks
C constant [0.34]
Subscripts and superscripts
rack rack unit
s supply air to IT equipment
c supply air from CRAC
other overheads. As shown in Fig. 1, the average PUE of South Korean
data centers is 2.66, which is much greater than that of global data
centers (1.7). In South Korea, the average PUE of commercial data
centers is 2.03, that of public data centers is 3.13, and that of certified
green data centers is 1.66 [3]. In addition, except for IT equipment,
cooling (38%) and power supply and distribution (9%) typically ac-
count for the major proportion of energy usage in a data center [4].
Therefore, these factors must be considered to reduce PUE. The ma-
jority of power supplied to IT equipment in a data center is converted to
heat, which must be removed to prevent IT equipment from over-
heating. Because at least thousands of IT devices are installed in each
data center, an enormous volume of cooling air is required to remove
heat. The goal of such a cooling system is to efficiently discharge
complex heat flows from the IT server room. The recent emergence of
next-generation IT equipment that operates under a high and variable
load has led to unexpected demands for existing cooling systems. Ac-
cordingly, cooling systems are suffering from excessive capacity and
inefficient operation, which makes it difficult to estimate the cooling
capacity that is actually required. To solve this problem, room-based,
row-based and rack-based cooling solutions have been developed. As
the latest IT devices are variable and tend to be densely installed, room-
based cooling systems cannot satisfy cooling requirements and are
subject to problems such as inefficiency, unpredictability and reduced
power density. The performance of the cooling system in a data center
depends on cooling type, power density per rack, and rack arrange-
ment. To overcome the limitations of existing room-based cooling sys-
tems and satisfy the demand for thermal comfort and energy efficiency,
this study proposes a new independent row-based cooling solution that
uses fully hot and cold aisle containment to compare and analyze the
thermal performance of this system with a comparison test against the
Fig. 1. Comprehensive PUE level for data centers and typical data center energy consumption.
2
Applied Thermal Engineering 169 (2020) 114857
h exhaust air form IT equipment to hall(room)
f supply air from raised floor to hall(room)
x xth IT server (average)
n negative pressure
bp by-pass
r recirculation
Max-rec maximum recommended/ASHRAE TC9.9 (2015)
Max-allowmaximum allowable/ASHRAE TC9.9 (2015)
Min-rec minimum recommended/ASHRAE TC9.9 (2015)
Min-allow minimum allowable/ASHRAE TC9.9 (2015)
SA supply air from CRAC/H unit
RA return air to CRAC/H unit
Equip IT equipment
existing room-based cooling. The main goal of this study is to evaluate
the improvement in data center cooling efficiency according to the IT
environment and energy use characteristics of the data center.
1.2. Literature review
For efficient cooling, the following factors should be considered: (1)
prevention of local overheating (hot spots), (2) cooling of the server and
not the entire server room, and (3) separation of hot and cold air. An
underfloor air distribution system, which is installed to improve the
cooling performance of a data center, always sets cold and hot aisles
together. Pantankar [5] showed that well-distributed air flow in the
lower part of a raised floor could prevent the mixing of cold and hot air
and the generation of hot spots in an IT server room. The author also
revealed that the internal pressure of a raised floor is the main factor
influencing air flow. In other words, the air flow is predominantly af-
fected by the height of the raised floor, the upper space of the perfo-
rated panel, and the obstacle arrangement beneath a raised floor. Karki
et al. [6] used an optimized one-dimensional computational model to
investigate the pressure distribution beneath a raised floor. Di-
mensionless variables were proposed as control factors for distributing
air flow. The results of air flow distribution, which were determined
using a one-dimensional model and could be compared with the di-
mensionless variables, were obtained as a comparative variable for a
three-dimensional model. Wang et al. [7] proposed perforated panels of
various sizes and applied them to a data center analysis model. Their
simulation results showed that the perforated panels improved the
temperature distribution without additional energy use. A modified
rack model was also proposed that decreased the area of cold aisles and
increased that of hot aisles. Compared to other data centers applying
J. Cho and J. Woo
the conventional cold aisle area, the inlet temperature of the rack de-
creased and the air mixing/recirculation and losses were remarkably
improved. Nada et al. [8] used a small-scale model including cold aisle
containment to prevent air mixing near the racks. The cooling perfor-
mance was examined by focusing on three alternative arrangements of
cold aisles: (1) open containment, (2) partial containment, and (3) full
containment. The rack inlet temperature of cold aisle containment was
reduced by 13–15.5% in the range of power density (load). Siriwardana
et al. [9] proposed a new technique of thermal load increase and dis-
tribution using a heat transfer CFD model with PSO (Particle Swarm
Optimization). The proposed model minimized the impact of the
thermal load increase in temperature distribution by simultaneously
considering the effects of increased thermal load and air flow dis-
tribution. Furthermore, Cho et al. [10] analyzed the air distribution
performance of six different cooling systems with and without duct
connections through CFD modeling. The above room-based cooling
solutions have satisfied data center requirements for a long time.
However, the latest IT equipment has created an unprecedented high-
density IT environment in data centers. For this reason, row-based so-
lutions have been proposed to solve the high-density heat load problem
of IT server rooms. In addition, difficulties in conducting simultaneous
experiments during real data center performance assessments has
caused difficulties in overcoming the errors and uncertainties resulting
from IT environment or the controls used during evaluations that in-
volve installations being in actual data centers and the demonstration of
long-term use. There were limited experimental works on comparing
two different systems under the same conditions at the same time for
securing the objectivity of judgment of thermal performance required.
2. Research methodology
In this study, a comparative assessment of the row-based and room-
based data center cooling systems was conducted through an on-site
measurement of a test data center. First, cooling efficiency evaluation
methods are investigated and quantitative evaluation indices are
Fig. 2. Research methodology and procedure.
3
Applied Thermal Engineering 169 (2020) 114857
analyzed, which reflect the unique characteristics of a data center.
Second, IT environment were analyzed through a measurement of air
temperature and related humidity distribution based on responses to
cooling system operation for existing room-based cooling system. The
main objective of step was evaluating IT cooling from local hot spot,
over-cooling and temperature imbalance. And then, the thermal per-
formance was examined by using six performance indices. Finally, the
room-based cooling system of a case study data center was renovated to
an independent row-based cooling system, on-site measurement of the
IT environment is conducted, and then the results are applied to a
performance index to objectively compare system efficiencies. Based on
this comparison, we establish an engineering basis for deriving the
performance and operating efficiency of each system. The limitations of
previous experimental studies are how to compare their performance of
row-based and room-based data center cooling systems accurately in
different situations. And how to reflect realities and secure reliability in
data center operation for simulation works. The differences and special
features of this experimental study is that two different forms of cooling
systems were evaluated based on the same location of the equipment
and the same cooling load in a same IT server room. With this ac-
complishment, reliability of comparison has made it possible to over-
come previous experimental researches. This study is practice-based
learning in a real data center renovation project. A series of field
measurements and performance analysis were carried out in the case
study data center. The aim of this study is to examine the air dis-
tribution performance relationships between CRAC unit and archi-
tectural design in IT server room. The research methodology and pro-
cedures are shown in Fig. 2.
3. An improved design of data center cooling
A data center cooling has two main functions: to provide a sufficient
cooling capacity and to distribute cooled air to each piece of IT
equipment. The first function is common to all cooling systems in-
cluding room-, row-, and rack-based systems. The total power load of
J. Cho and J. Woo
the IT equipment is cooled in the form of thermal load. The main dif-
ference among the three cooling systems is related to the second
function, that is, the distribution of cooled air to the IT equipment.
Unlike power distribution, air flow cannot be controlled. Power supply
can be controlled in a visualized form of wired power lines; however,
while air distribution can be somewhat controlled by the design of an IT
server room, the air flow cannot be observed and the spatial config-
uration has a substantial influence. Each cooling system has been
continuously developed for the main purpose of controlling air dis-
tribution.
3.1. Room-based cooling design
In a room-based cooling system, multiple CRAC units are simulta-
neously operated to set the unit of an IT server room and treat the
thermal load of an entire space. The room-based cooling system can
either directly supply cooled air from at least two indoor CRAC units
without control using ducts, dampers or diffuser, or partially control air
supply and return using the path of a room space. The design of air
distribution depends significantly on the specific project characteristics.
A large-scale data center uses a raised floor to distribute air to hot /cold
aisles that are arranged to create efficient air flow for IT equipment and
racks. Designs based on room-based cooling systems are largely affected
by IT room constraints (ceiling height, room spatial form, rack ar-
rangement, location of CRAC unit, and power distribution for each
piece of IT equipment). When the power density increases and cold/hot
aisle containment is not applicable, it is particularly difficult to achieve
uniform cooling performance for the entire IT server room. Moreover, if
a new IT environment is created by moving, adding, or modifying IT
equipment, the existing analysis model becomes invalid and thus new
analyses and tests are required. In particular, when obtaining a backup
system of CRAC, analysis becomes very complicated and verification is
difficult. Fig. 3 shows the configuration of a room-based cooling system.
If the rated capacity of the CRAC unit increases and containment is not
applied, the room-based cooling system may exhibit reduced cooling
performance. By-pass occurs when the majority of cooled air distributed
by the CRAC unit does not reach the IT equipment but instead returns to
the CRAC unit. In this case, despite an adequate rated capacity, the
cooling requirements of some regions may exceed the cooling capacity
of the CRAC unit. Moreover, the internal temperature of the server
increases due to the re-circulation of hot air removed from the IT
equipment by the heat removal process; thus, the error rate also in-
creases.
Fig. 3. Room-based cooling design; (a) the basic concept of room-based cooling and (b) example of a traditional room-based cooling.
4
Applied Thermal Engineering 169 (2020) 114857
3.2. Row-based cooling design
In many legacy data centers with row-based cooling system, CRAC
units constitute an exclusive cooling system for selected rack rows.
CRAC units can be installed either between racks. The row-based
cooling system sets a shorter and clearer air distribution path than the
conventional room-based cooling system. Moreover, as air distribution
and flow are easy to predict, the rated capacity of the CRAC unit can be
fully exploited, enabling a response to higher power density. Aside from
its cooling performance, the row-based cooling system has many ad-
vantages. As the transfer path of the cooling air can be shortened, the
power consumption of the CRAC unit fan can also be reduced, which
results in higher efficiency. Fan operation of the cooling system occu-
pies a high percentage of energy use in a data center and the room-
based cooling system operates a large capacity CRAC unit because
uninstalled IT equipment is also considered in the expansion potential
of the data center. Accordingly, inefficiency becomes a problem.
Designs based on the row-based cooling system can be allocate the right
cooling capacity and redundancy required in a specific zone (row). In
addition, a specific row can be set redundancy of N + 1 or 2 N cooling
system. If constructing a data center with a high power density (10 kW
or above per rack), the row-based cooling system is more advantageous
and no raised floor is required, resulting in a more flexible spatial
configuration. The cooling system shown in Fig. 4 can be configured to
have cold/hot aisle containment to improve its power density. As such a
configuration does not mix hot and cold air; the cooling performance
can be accurately predicted. Additionally, because the layouts of row-
based cooling systems are standardized and simplified, a modular de-
sign is possible.
3.3. Improving legacy data center cooling by using an independent row-
based system
Fig. 3(a) and Fig. 4(a) show the basic configurations and concepts of
the three cooling systems, whereby a CRAC unit is connected to an IT
server installed in a rack [11]. Although a CRAC unit may have a dif-
ferent physical layout, the CRAC unit of a room-based cooling system
controls an entire IT server room, that of an independent row-based
cooling system controls part of a section or row (group) where racks are
arranged. The legacy data center cooling strategy employs a room SA
and room RA system. As shown in Fig. 5(a), this cooling strategy may
not efficiently remove the heat concentrations of IT server at specific
locations due to air by-pass and recirculation in the room. In short, SA is
J. Cho and J. Woo
Fig. 4. Row-based cooling design; (a) the basic concept of row-based cooling and (b) example of an in-row modular cooling system.
in flowed near inlets of IT equipment, but RA is not captured near
outlets of IT equipment. There will be an expected high risk of IT
downtime from hot spots caused by overheating in an IT server room.
An improved row-based cooling system in Fig. 5(b) as the best ar-
rangement is to RA of in-row CRAC units near IT outlets at the hot aisle
containment, and to SA of CRAC units at the cold aisle containment.
This cooling strategy has been adopted in some data centers, but the full
cold/hot aisle containment is not installed. Because the location of the
cold aisle and hot aisle is very important, full cold/hot aisle contain-
ment coupling with in-row CRAC units was developed for effective
removal of IT heat load. The overall goal of this study is to find an
improved cooling system based on IT heat concentrations which are
measured parameter and best design practices should be followed.
3.4. Evaluation method for the cooling efficiency of a data center
Many researchers have proposed evaluation indices for data center
cooling efficiency. Xie et al. [12] reviewed existing thermal environ-
ment evaluation metrics are reviewed at room level and rack level re-
spectively. Table 1 presents the main indices for evaluating the cooling
performance of a data center. Sharma et al. [13] proposed the SHI
(Supply Heat Index) and RHI (Return Heat Index) to evaluate the effi-
cient use and re-circulation of cooling air in a cooling path. RHI and SHI
indicate the mixing of input cool air flow to the rack and output hot air
flow from the rack. Q is the total heat propagation of racks and in-
dicates the amount of cool air enthalpy increase before its entering to
the rack. Where mirack
, j i s the entrance flow to the ith rack in the jth row.
Usually RHI > 0.8 and SHI < 0.2 is acceptable respectively. As much
Fig. 5. Data center cooling strategies of a reference data center; (a) an existing room-based cooling system and (b) an improved independent row-based cooling
system.
5
Applied Thermal Engineering 169 (2020) 114857
SHI decreases, RHI will increase and the design will approach to the
ideal design of cooling system for data center. This condition shows less
mixing of cool air with the output hot air from the rack. However, both
SHI and RHI are based on room-level units and thus are not effective for
units of IT equipment. β is another evaluation index based on room-
level units that provides information about overheating to analyze the
re-circulation of cooling air [12]. Herrlin [14] presented the RCI (Rack
Cooling Index) to measure the cooling performance of IT equipment,
which is divided into RCIHi and RCILo according to upper and lower
temperature limits. RCI more than 96% is desirable. The 100% value
shows that all racks have cooled by a specific standard. The ASHRAE
guideline [15] considers RCI to be very important as this index uses
units of IT equipment to evaluate the thermal performance of IT
equipment. RCIHi is a scale of over temperature absence. 100% value
represents that over temperature does not occur. As much as it de-
creases, the probability of over temperature increases. RCIHi is, the
complement of RCILo, the most significant index in designing the plant
of data center. Increasing the rate of input air to the racks improves this
index and raises the energy consumption. Herrlin [16] proposed the RTI
(Return Temperature Index) to explain air re-circulation and by-pass.
The ideal condition of RTI is a 100% match between the supply and
utilization of cooling air. Any RTI below 100% indicates some by-pass
of cooling air returning to the CRAC/H unit without being used. On the
other hand, any RTI above 100% indicates re-circulation of cooling air.
The ideal percentage for this index is 100% and RCI never achieves
100% in practice. However, the desired value would be feasible by aisle
containment. Tozer et al. [17] proposed indices to explain the dis-
tribution of cooling air in a data center by utilizing heat transfer media
 J. Cho and J. Woo

Table 1
Evaluation metrics of data center thermal management [12].
Metrics Information Air distribution Formula Rating

H/F R/F

SHI Recirculation degree in air mixing Yes Yes Q Q= mirack Tc ] Target 0

j i ,j [(Thrack )i, j (Tsrack )i,j ] Q = j i mirack
,j [(Tsrack )i,j
Q+ Q Good <0.2
RHI Cooling effect of cold air supply Yes Yes Q Target 1
Q+ Q Good >0.8
RCILo Rack cooling condition according to cold threshold values Yes Yes (TMin rec Tx )Tx < TMin rec Ideal 100%
1
(TMin rec TMin allow) × n Good ≥96%
Acceptable 91%~95%
Poor ≤90%
RCIHi Rack cooling condition according to hot threshold values Yes Yes (Tx TMax rec )Tx > TMax rec Ideal 100%
1
(TMax allow TMax rec ) × n Good ≥96%

6
Acceptable 91%~95%
Poor ≤90%
RTI Extend of bypass and recirculation Yes Yes TRA TSA Re-circulation >100%
TEquip By-pass <100%
Acceptable >100 ± 5%>
Poor <100 ± 30%<
β Recirculation and overheating extent Yes Yes Ts Tc Target 0
T h Ts
NP Extend of negative pressure airflow into under floor No Yes mn Tf Tc –
=
mc Tr Tf
BP Extend of bypass airflow with data center No Yes mbp Th Tr Ideal 0
=
mf Th Tf Good <0.05
Acceptable 0.05 ~ 0.2
R Extend of recirculationairflow in cold aisles No Yes mr Ts Tf Target 0
=
ms Th Tf Good <0.2
BAL Difference between cooling air supplied and demand No Yes mc Th Ts Ideal 1
=
ms Tr Tc Good <0.85
CI Airflow ingested by local component Yes Yes mh mc Target 0
CIcoldaisle = CIhotaisle =
mf ms
Applied Thermal Engineering 169 (2020) 114857
J. Cho and J. Woo
and heat balance equations. Here, the ratios of three air mixing pro-
cesses are measured: NP (Negative Pressure), BP (By-pass), and R
(Recirculation). BAL (Balance) explains the difference between cooling
air supplied from the CRAC/H and the quantity required by the IT
server. In addition, Vangilder [18] introduced the CI (Capture Index)
based on the flow rate of the heat transfer medium. CI is a row-based
indicator that measures the efficiency of air distribution attributable to
a cooling unit or fan that can be a cooling path. PUE (Power Usage
Effectiveness) is currently viewed as the industry-preferred metric for
measuring infrastructure energy efficiency in data centers. PUE is de-
fined as the ratio of total facilities energy to IT equipment energy [19].
Data center experts were thinking about the relationship between the
power consumed by the IT hardware itself and the total power absorbed
by the containment facility. There is no ways to focus on the PUE to
improve 'energy efficiency' while neglecting total energy consumption.
While this caution is useful and valid, it is very difficult to prove PUE's
potential value to each individual facility seeking to improve its own
performance. PUE is not very well suited for evaluating the energy
consumption at their partially modified cooling system. PUE is not an
efficiency metric for data center infrastructure as this infrastructure
involves the underlying reason for the data center, that being the in-
formation technology components. The PUE metric has driven energy
efficiency gains in the areas of thermal management, cooling and power
distribution in the data centers, but it is not a true efficiency measure in
proper engineering terms [20,21].
Fig. 6. Reference data center for the experimental study.
7
Applied Thermal Engineering 169 (2020) 114857
4. Full-scale experimental study
4.1. Experimental procedure
This study is practice-based learning in a data center project. The
results of temperature/humidity distribution experiments help to un-
derstand the thermal performance of IT environments for example in
“S” data center (in Korea). This data center decided to permit an in-
dependent row-based cooling system for IT server rooms in 2018 ac-
cording to the preliminary cooling strategies study. A series of full-scale
field measurements were conducted. Field measurement methods used
to analyze data can be classified into two categories as IT power and IT
environment.
4.2. Reference data center
In Fig. 6, a field experiment for improving the cooling performance
of a medium-scale reference data center (GFA: 735 m2) was performed.
Three out of four floors contained server rooms, and a room-based
cooling system was currently in operation. Measurement of the IT en-
vironment was limited to a zone of the IT server room on the third floor.
This was because the room-based cooling system was due to be replaced
with a row-based cooling system to comparatively evaluate the cooling
performance. Fig. 7 shows the data center under analysis, the mea-
surement zone of the IT environment on the third floor, and the sensor
locations. Originally, IT server rooms adopted the cooling system with
hard floor where cold air was distributed directly from the CRAC unit.
J. Cho and J. Woo Applied Thermal Engineering 169 (2020) 114857

Fig. 7. On-site temperature and humidity measurement points (IT environment) of an existing room-based cooling at the IT server room #2.

4.2.1. Measurements of a room based cooling
Air temperature, humidity and input IT power were measured at
rack (A), located far from the CRAC unit and at another rack (B), lo-
cated near the CRAC unit. The ten measurement points included the
upper and lower inlets (4), the upper and lower outlets (4), the SA
(supply air) of the CRAC unit, and the RA (return air) form IT room. As
shown in Fig. 8, a single unit rack consisted of eight IT servers, the
physical specifications of which are shown in Table 2. Each IT server
had a power consumption of 870 W and a fan air flow rate of 258 m3/h.
The cooling load was at least 95% of the power consumption. Each rack

Fig. 8. Rack server configurations of the IT equipment.

8
J. Cho and J. Woo Applied Thermal Engineering 169 (2020) 114857

Table 2 4.2.3. Comparative measurement method

Detailed specifications of the IT equipment. In a comparative measures design, the scores in the two cooling
IT equipment Server rack systems are paired because they come from the same conditions. In
other words, each cooling system is tested in same experimental con-
Quantity 8 (EA/rack) 6 (EA/zone) dition. This technique can be an excellent method to reduce random
Size – 600x600x1800mm
error between systems that is due to environment differences. We
Power 870 (W/equip.) 7.0 (kW/rack)
Heat load 830 (W/equip.) 6.7 (kW/rack)
carefully considered each factor and be sure that both cooling systems
Fan airflow rate 258 (m3/h) 2,064 (m3/h) have the same experimental setting such as same IT equipment and IT
Real airflow rate 232 (m3/h) 1,858 (m3/h) heat load. The room-based and row-based cooling systems are two
Total heat load 6.7 (kW/rack) 40.2 (kW/zone) different approaches, which indicate that the cooling systems should be
Total airflow rate 1,858 (m3/h·rack) 11,148 (m3/h·zone)
used under their suitable conditions, on the same location.
Required airflow 277 (m3/h·kW) –
Remark – 11 usRT
4.2.4. Measuring instruments accuracy
Measurements were continuously conducted for 3 days (72 h) every
had a cooling load of approximately 7.0 kW/rack and the required air
10 min using TR-72WF/THA-3001 measuring equipment. There are
flow of 2,000 m3/h. five rack servers were applied to the new row-
two type of evaluation of uncertainty. Type A is method of evaluation of
based cooling system, which required a cooling capacity of 35 kW.
uncertainty by the statistical analysis of series of observations and type
B is method of evaluation of uncertainty by means other than the sta-
tistical analysis of series of observations [22]. Accuracy with mea-
4.2.2. Measurements of a row-based cooling
surement instruments is very important in the type B. All instruments
A part of the room-based server cooling system was replaced with a
have been calibrated with certificates for successfully completed test.
row-based cooling system to compare the resulting IT environment. The
Table 3 contains the list of instruments with measuring precision and
cooling performance was evaluated both before and after measurement.
range of measured parameters. Accuracy is the degree to which the
As shown in Fig. 9, containments were installed in both the cold aisle
measured value agrees with the true value.
and hot aisle to distinguish the zone where five rack servers were ar-
ranged and three small-capacity CRAC units were installed. The tem-
perature and humidity of the inlet and outlet of each server rack were 4.3. Measurement results
measured at the same locations as for the room-based cooling system at
10 points including the SA (cold aisle) and RA (hot aisle) of a re- 4.3.1. IT environments of a room-based cooling
presentative CRAC unit. Table 3 and Fig. 10 show the IT environment of the room-based
server cooling system from Sep 4 to Sep 7, 2018. The average SA

Fig. 9. On-site temperature and humidity measurement points (IT environment) of a new row-based cooling at the same IT server room #2.

9
J. Cho and J. Woo
Table 3
Measuring instruments and measurements range.
Sensor Channels Units Range Accuracy Resolution
Thermistor Temperature °C 0 to 55 °C ±0.5 °C 0.1 °C
1ch
Polymer Humidity % RH 10 to 95% ±5 %RH 1 %RH
Resistance 1ch RH at 25 °C,
50 %RH
temperature of the CRAC unit (T#10) was 15.8 °C and the average
temperature of air inlets from the cold aisle to the rack servers (T#01,
T#02, T#05, T#06) was 24.1 °C which satisfied the recommended
range of ASHRAE Class A-1 [15] (DB 18–27 °C). A loss of at least
ΔT = 8.3 °C was measured along the distribution path (Fig. 11(a)). In
addition, the average temperature of air outlets (T#03, T#04, T#07,
T#08) to the hot aisle after removing heat in the rack server was
39.9 °C. In other words, a change in temperature, ΔT = 15.9 °C was
maintained. However, the average RA temperature of the CRAC unit
(T#09) was 24.0 °C, indicating serious mixing of air in the room
(Fig. 11(b)). As the temperature distribution was not affected by ex-
ternal factors such as building skin load but mostly by the thermal load
of the equipment, it remained steady. Nevertheless, as the average
temperature of the cold aisle was higher than the average RA tem-
perature, the air of the hot aisle was recirculated or mixed substantially
in many parts. Relative humidity does not have a significant effect on
cooling efficiency; however, as the IT equipment may be affected by
low relative humidity, this parameter should still be considered. A re-
lative humidity of approximately 35% was maintained in the cold aisle,
which satisfied the permissible range of ASHRAE Class A-1 [15] (RH
8 ~ 80%). However, when heat was removed and temperature in-
creased in the hot aisle, the relative humidity was approximately 15%.
Accordingly, humidification could be considered if the relative hu-
midity decreased further.
4.3.2. IT environments of a row-based cooling
After measuring of IT environments for an existing room-based
cooling, fully cold/hot aisle containments and in-row CRAC units had
been installed for an independent row-based cooling system on same
white space. Table 5 and Fig. 12 indicate the IT environment of the new
Fig. 10. Air temperature distributions around IT equipment for the room-based cooling system; (a) inlet air and (b) outlet air.
Applied Thermal Engineering 169 (2020) 114857
Responsiveness Dimensions Operating Environment
Response Time (90%): H58 mm × W78 mm × D26 mm Temperature: −10 to 60 °C
Approx. 7 min. Humidity: 90%RH or less (no
condensation)
room-based cooling system from Nov 1 to Nov 4, 2018. To maintain the
cold aisle of the row-based cooling system at 25–28 °C, three CRAC
units were set to start and stop and the fan was set to operate con-
tinuously. The average SA temperature of the CRAC unit (T#10) was
21.2 °C and the average temperature of air flowing from the cold aisle
to the rack servers (T#01, T#02, T#05, T#06) was 25.8 °C which sa-
tisfied the allowable range of ASHRAE Class A-1 [15] (DB 15–32 °C).
Thus, approximately ΔT = 4.6 °C was lost along the distribution path
(Fig. 13(a)), making this system at least 50% more effective than the
room-based cooling system. In addition, the average temperature of air
(T#03, T#04, T#07, T#08) emitted to the hot aisle after heat removal
in the rack server was 41.5 °C. In other words, approximately
ΔT = 15.7 °C was maintained, which was similar to the room-based
cooling system. Moreover, the average RA temperature (T#09) of the
CRAC unit was 33.8 °C, indicating that mixing of air in the cold aisle
was significantly improved (Fig. 13(b)). As the temperature of the cold
aisle was at least 10 °C less than the RA temperature, the mixing of
recirculated air in the hot aisle was also improved. The distribution of
relative humidity was similar to that of the room-based cooling system.
4.3.3. Assurance of same experimental conditions by IT power
The major heat gain in a data center is from the servers. For a server,
power input typically dissipates heating or, in other words, require
cooling. To assure the same experimental conditions between room-
based cooling and row-based cooling, real-time IT power uses were
measured at rack (A) and rack (B) by power distribution units (PDUs)
from Sep 1 to Sep 30 and from Nov 1 to Nov 30, 2018. The look-back
measurement method is the preferred method of tracking the IT heat
loads (cooling loads) variable IT equipment worked, and is referred to
as the IT input power. Fig. 14 shows the IT input power distributions at
10
J. Cho and J. Woo Applied Thermal Engineering 169 (2020) 114857

Fig. 11. Box plot of IT environment data of the room based cooling system; (a) cold zone and (b) hot zone.

rack (A) and rack (B), respectively. When the rack (A) and rack (B) of
the new row-based cooling system were same location of existing room-
based cooling, the mean IT input power are very constant from Sep 1
to Nov 30. This shows that cooling load was also constant and a con-
tinuous heat transfer effect caused by the heat of the IT equipment in
the IT server room. Based on the IT cooling load, the experimental
conditions of two cooling strategies were verified the same.
5. Evaluation of cooling efficiency
5.1. Thermal performance of a room-based cooling
In the above (3.4) section, various metrics for evaluating the effi-
ciency of cooling air distribution were investigated. The analysis of IT
environmental factors (temperature) through on-site measurement is
the key for deriving a quantitative index using evaluation metrics for
the cooling efficiency of a data center. The data center used in this
study did not adopt a raised floor for air supply in the room-based
cooling system of the server room. Accordingly, it was not possible to
evaluate the efficiency of every metric surveyed above. For this reason,
SHI, RHI, RCI, RTI, and β, all of which could be calculated, were ana-
lyzed. Fig. 15 shows the continuous distribution of performance metrics
during measurement of the IT environment. As presented in Table 6, the
cooling efficiency was analyzed using objective metrics or based on the
criteria and results of each metric. SHI and RHI, which are di-
mensionless metrics with respect to the mixing of air emitted from the
IT equipment and low temperature cooling air and heat loss, are com-
plementary to each other. The distribution of SHI was 0.31–0.37
(Target: 0.0), and that of RHI was 0.63–0.69 (Target: 1.0). Thus, neither
index exhibited appropriate performance. On the other hand, RCI
maintained an appropriate temperature within the permissible tem-
perature range of ASHRAE Class A-1 [15] for a server room. RCILo
evaluates whether an adequate thermal environment is maintained
within the recommended temperature range (18–27 °C) and the range
of allowable temperature (15–32 °C). An optimal condition is
maintained within an 100% permissible temperature range in every
inlet. RCIHi, which represents the opposite concept, evaluates whether
an appropriate thermal environment is maintained within the lower
range. The permissible temperature is maintained in the range of
89.3–100%. This level causes no specific problems to the thermal en-
vironment. RTI can measure the energy performance of a cooling
system. The ratio between the temperature rise in SA and RA of the
CRAC/H unit and that of the cooling air with heat removed after the IT
equipment is the evaluation criterion. RTI was 47–56%, indicating
substantial by-pass. In other words, cooling air from the CRAC/H unit
could not pass through the IT equipment but was instead returned to
the CRAC/H unit. More air than required by the rack server was sup-
plied and a large portion was lost, which resulted in energy inefficiency.
β is also used to determine recirculation and overheating; an in-
efficiency range of 45–60% was maintained. Based on the above in-
dices, the room-based cooling system of the data center showed loss in
the cooling distribution path, mixing of cooling air, and inefficient air
by-pass. In addition, significant inefficiency resulted from the need to
supply additional cooling energy to maintain an appropriate IT en-
vironment for RCI.
5.2. Thermal performance of a row-based cooling
As for the existing room-based cooling system, the cooling efficiency
of the new row-based server cooling system was analyzed using SHI,
RHI, RCI, RTI, and β. Fig. 16, Table 7 show the continuous distribution
of performance metrics during measurement of the IT environment. As
presented in Table 8, the cooling efficiency was analyzed using objec-
tive metrics or the criteria and results of each metric. The average SHI
and RHI were 0.22 and 0.78, respectively. Both results approximated
the appropriate performance and indicated improvements of 37.1% and
20.0%, respectively, compared to the room-based cooling system. RCI
also maintained an adequate temperature despite the increase of inlet
temperature in the CRAC unit. RCILo maintained an optimal condition
within a 100% permissible temperature range in all air inlet of IT

Table 4
Descriptive statistics of IT environment of the room-based cooling system: air temperature and relative humidity.
Measuring Rack A Rack B Return Air Supply Air
Points (Room air) from CRAC
Average Inlet Outlet Inlet Outlet
IT environment (upper/lower) (upper/lower) (upper/lower) (upper/lower)

T#01 T#02 T#03 T#04 T#05 T#06 T#07 T#08 T#09 T#10

Air temperature (°C) Mean 25.68 23.60 37.10 43.42 23.87 23.33 38.62 40.41 23.97 15.79
Min 25.00 23.20 36.50 42.90 23.20 23.00 38.10 39.90 23.20 14.70
Max 27.70 25.10 39.10 45.10 25.30 24.80 40.60 42.50 26.00 17.00
StDev 0.348 0.271 0.363 0.310 0.271 0.229 0.369 0.373 0.460 0.542

Relative humidity (%) Mean 33.46 38.82 16.39 11.04 35.94 39.17 15.02 12.78 37.23 64.47

11
J. Cho and J. Woo Applied Thermal Engineering 169 (2020) 114857

Table 5
Descriptive statistics of IT environment of the new row-based cooling system: air temperature and relative humidity.
Measuring Points Average IT environment Rack A Rack B Return Air (Hot Supply Air (Cold
aisle) aisle)
Inlet (upper/lower) Outlet (upper/lower) Inlet (upper/lower) Outlet (upper/lower)

T#01 T#02 T#03 T#04 T#05 T#06 T#07 T#08 T#09 T#10

Air temperature (°C) Mean 24.05 26.22 37.96 42.15 26.40 26.44 41.13 44.73 33.81 21.24
Min 20.35 24.30 37.20 41.42 22.70 24.20 40.50 43.90 33.10 20.80
Max 27.15 29.10 38.90 42.87 29.50 28.30 41.80 45.50 34.80 21.80
StDev 1.780 1.299 0.307 0.265 1.780 1.140 0.273 0.285 0.335 0.222

Relative humidity (%) Mean 32.14 35.15 17.04 11.98 34.11 34.48 14.15 11.96 21.22 36.52

Fig. 12. Air temperature distributions around IT equipment for the new row-based cooling system; (a) inlet air and (b) outlet air.

Fig. 13. Box plot of IT environment data of the new row-based cooling system; (a) cold zone and (b) hot zone.

servers. RCIHi, which is the opposite concept, decreased slightly in the
average lower range of 96.3% but maintained an appropriate tem-
perature. The average RTI was 90%, which indicated some by-pass but
was within an appropriate range for the operating environment. This
value was also a 73.2% improvement over that of the room-based
cooling system. Although the containment was applied to both cold and
hot aisles, the airtightness was not high enough; thus, some of the
cooling air supplied by the CRAC/H unit did not pass through the IT
equipment but was instead returned to the unit. β, which was used to
judge recirculation and overheating, was 29.4% on average, which
approximated an efficient range and also a 44.3% improvement over
that of the room-based cooling system. (Fig. 17)
By analyzing the cooling efficiency of the row-based cooling system
with full containment of cold and hot aisles using the six indices, losses
in the distribution path and the mixing and by-pass of cooling air can be
minimized and the inlet temperature of the CRAC unit can be main-
tained at a high level. Thus, the equipment efficiency (e.g. COP) of the
cooling system can be effectively improved. Moreover, if a small-ca-
pacity fan with a low static pressure replaces the large-capacity, high-
static pressure fan, the power consumption of the fan can be effectively

12
J. Cho and J. Woo Applied Thermal Engineering 169 (2020) 114857

Fig. 14. IT input power distributions of Rack A and Rack B; (a) Sep 1 to Sep 30, 2018 and (b) Nov 1 to Nov 30, 2018.

Fig. 15. Effect of the uncontained layout on the evaluation metrics for the room-based cooling; (a) SHI/RHI, (b) RCIHi/RCILo, and (c) RTI/β.

Table 6
Results of evaluation metrics for thermal management of the room-based cooling system.
Metrics SHI RHI RCILo/RCIHi RTI β

Rating Target Target Ideal Acceptable Recirculation Acceptable Target

0 1 100% 91%~95% >100% >100 ± 5%> 0
Good Good Good Poor By-pass Poor
<0.2 >0.8 ≥96% ≤90% <100% <100 ± 30%<

Evaluation Max 0.37 0.69 100% 100% 56.3% 59.6%

Min 0.31 0.63 100% 89.3% 46.5% 45.1%
StDev 0.0128 0.0128 0.00 1.28 2.23 2.97

Results Mean 0.35 0.65 RCILo: 100%(Ideal) RCIHi: 97.5%(Good) By-pass: 51.9%(Poor) 52.8%

13
J. Cho and J. Woo Applied Thermal Engineering 169 (2020) 114857

Fig. 16. Effect of the uncontained layout on the evaluation metrics for the row-based cooling; (a) SHI/RHI, (b) RCIHi/RCILo, and (c) RTI/β.

Table 7
Results of evaluation metrics for thermal management of the row-based cooling system.
Metrics SHI RHI RCILo/RCIHi RTI β

Rating Target Target Ideal Acceptable Recirculation Acceptable Target

0 1 100% 91%~95% >100% >100 ± 5%> 0
Good Good Good Poor By-pass Poor
<0.2 >0.8 ≥96% ≤90% <100% <100 ± 30%<

Evaluation Max 0.33 0.90 100% 100% 107.9% 50.3%

Min 0.10 0.67 100% 87.0% 74.8% 11.1%
StDev 0.051 0.051 0.00 3.66 6.59 8.63

Results Mean 0.22 0.78 RCILo: 100%(Ideal) RCIHi: 96.3%(Good) By-pass: 89.9%(Acceptable) 29.4%

Table 8
The thermal performance percentage difference of two cooling systems.
Metrics SHI RHI RCILo RCIHi RTI β

Existing room-based cooling 0.35 0.65 100% 97.5% 51.9% 52.8%

New row-based cooling 0.22 0.78 100% 96.3% 89.9% 29.4%
Percentage difference −37.1% 20.0% – −1.2% 73.2% −44.3%

reduced. cooling system in an experimental measurement, comparison analyzing

6. Conclusion
Main purpose of this research is performance analysis on two data
center cooling strategies, which is the most important for suitable IT
environment. This study investigated the thermal performance of an
existing room-based cooling system and a new independent row-based
the thermal environment based on the inlet and outlet air temperature
distribution of IT equipment. To achieve this goal, the existing room-
based cooling system of a reference data center under analysis was
changed to a new independent row-based cooling system basis on same
IT server room. Quantitative evaluation indices, which reflect the
characteristics of the data center, were used. Results of the performance
analysis were verified by comparison with the results that achieved

14
J. Cho and J. Woo
Fig. 17. Box plot of thermal performance comparison of two cooling system; (a) SHI/RHI, (b) RCIHi/RCILo, and (c) RTI/β.
through an analysis using the same conditions as the experimental
measurement. A verified experimental condition was then used to
analyze the changes in thermal performance while changing the cooling
systems. The research results can be summarized as follows;
(1) The IT environment of the room-based server cooling system was
measured in a medium scale reference data center. Six evaluation
indices were analyzed based on the air temperature and relative
humidity of the inlet/outlet of IT equipment and CRAC units.
(2) The average SA temperature of the CRAC unit of an existing room-
based cooling and a new independent row-based cooling were
15.8 °C and 21.2 °C, and the average temperature of air inlets of the
IT severs (T#01, T#02, T#05, T#06) of two cooling system were
24.1 °C and 25.8 °C, respectively, which satisfied the allowable
range of ASHRAE Class A-1. Thus, ΔT = 8.3 °C and ΔT = 4.6 °C
were lost along the distribution path, making the new row-based
cooling at least 50% more effective than the existing room-based
cooling system.
(3) When the rack A and rack B of the new row-based cooling system
15
Applied Thermal Engineering 169 (2020) 114857
were same location of existing room-based cooling, the mean IT
input power were very constant and a continuous heat transfer ef-
fect caused by the heat of the IT equipment in the IT server room.
Based on the IT cooling load, the experimental conditions of two
cooling strategies were verified the same.
(4) The cooling energy efficiency was analyzed using six indices (SHI,
RHI, RCI, RTI, and β). The room-based cooling system exhibited
inefficient operation such as losses in the distribution path and the
mixing and by-pass of cooling air. Thus, the room-based cooling
system was more inefficient as more cooling energy was required to
maintain an appropriate IT environment.
(5) For the fully containment of cold and hot aisles, the efficiency of the
row-based cooling system was better than that of the room-based
cooling system. In other words, SHI and RHI were improved by
37.1% and 20.0%, respectively, and RTI and β were improved by
73.2% and 44.3%, respectively.
(6) A comprehensive and objective analysis of the operational factors
that influence cooling efficiency and the impact of each factor can
reduce the cooling energy of a high-density data center and allow a
J. Cho and J. Woo
comparison of the cooling performance of room-based and row-
based systems.
The chosen data center cooling system has an influence on the inlet
air temperature of IT equipment and air distribution efficiency in the IT
server room. An independent row-based cooling is the improved system
with less risk of recirculation and bypass from CRAC units to IT servers.
These findings are expected to provide important evidence that can aid
in the development of design strategy for effective removal of heat load
in high-density data centers. In future work, a sensitivity analysis
should be performed for the decrease in cooling energy at various inlet
temperatures.
Author contributions
Jinkyun Cho: substantial contribution to conception and design,
substantial contribution to analysis and interpretation of data, drafting
the article, critically revising the article for important intellectual
content, final approval of the version to be published. Jesang Woo:
substantial contribution to acquisition of data.
Acknowledgement
This research was supported by grant of the research fund of the
MOTIE (Ministry of Trade, Industry and Energy) in 2018. Project
number: 20182010600010.
This research was supported by newly appointed professor research
fund of Hanbat National University in 2019.
Appendix A. Supplementary material
Supplementary data to this article can be found online at https://
doi.org/10.1016/j.applthermaleng.2019.114857.
References
[1] J. Cho, J. Yang, W. Park, Evaluation of air distribution system's airflow performance
for cooling energy savings in high-density data centers, Energy Build. 68 (2014)
270–279.
[2] V. Avelar, D. Azevedo, A. French, PUETM: a comprehensive examination of the
16
Applied Thermal Engineering 169 (2020) 114857
metric, White paper #49, the Green Grid Technical Committee, 2012.
[3] Korea IT Service Industry Association, A foundation research for the data center
industry development, ICT & Future Planning of Korea, Ministry of Science, 2015.
[4] J. Cho, J. Yang, C. Lee, J. Lee, Development of an energy evaluation and design tool
for dedicated cooling systems of data centers: Sensing data center cooling energy
efficiency, Energy Build. 96 (2015) 357–372.
[5] S.V. Patankar, Airflow and cooling in a data center, J. Heat Transfer 132 (7) (2010)
73001–73017.
[6] K.C. Karki, S.V. Patankar, Airflow distribution through perforated tiles in raised
floor data centers, Build. Environ. 41 (2006) 734–744.
[7] I.N. Wang, Y.Y. Tsui, C.C. Wang, Improvements of airflow distribution in a con-
tainer data center, Energy Procedia 75 (2015) 1819–1824.
[8] S.A. Nada, K.E. Elfeky, Experimental investigations of thermal managements solu-
tions in data centers buildings for different arrangements of cold aisles contain-
ments, J. Build. Eng. 5 (2016) 41–49.
[9] J. Siriwardana, S.K. Halgamuge, T. Scherer, W. Schott, Minimizing the thermal
impact of computing equipment upgrades in data centers, Energy Build. 50 (2012)
81–92.
[10] J. Cho, T. Lim, B.S. Kim, Measurements and predictions of the air distribution
systems in high compute density (Internet) data centers, Energy Build. 41 (2009)
1107–1115.
[11] K. Dunlap, N. Rasmussen, Choosing Between Room, Row, and Rack-based Cooling
for Data Centers. White paper #130, Schneider Electric’s Data Center Science
Center, 2012.
[12] M. Xie, J. Wang, J. Liu, Evaluation metrics of thermal management in data centers
based on exergy analysis, Appl. Therm. Eng. 147 (2019) 1083–1095.
[13] R.K. Sharma, C.E. Bash, C.D. Patel, Dimensionless parameters for evaluation of
thermal design and performance of large scale data centers, American Institute of
Aeronautics and Astronautics (AIAA), 2002, pp. 3091–3101.
[14] M.K. Herrlin, Rack cooling effectiveness in data centers and telecom central offices:
the rack cooling index (RCI), ASHRAE Trans. 111 (Part 2) (2005) 725–731.
[15] ASHRAE TC 9.9, Thermal Guideline for Data Processing Environments, American
Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc., Atlanta, GA,
2015.
[16] M.K. Herrlin, Improved data center energy efficiency and thermal performance by
advanced airflow analysis, Proceedings of Digital Power Forum, 2007, pp. 10–12.
[17] R. Tozer, C. Kurkjian, M. Salim, Air management metrics in data centers, ASHRAE
Trans. 115 (2009) 63–70.
[18] J.W. Vangilder, S.K. Shrivastava, Capture index: An airflow-based rack cooling
performance metric, ASHRAE Trans. 113 (2007) 126–136.
[19] V. Avelar, D. Azevedo, A. French, PUE: A Comprehensive Examination of the
Metric, White paper #49, the Green Grid, 2012.
[20] J. Yuventi, R. Mehdizadeh, A Critical Analysis of Power Usage Effectiveness and Its
Use as Data Center Energy Sustainability Metrics, CIFE Working Paper #WP131,
Stanford University, 2013.
[21] G.A. Brady, N. Kapur, J.L. Summers, H.M. Thompson, A case study and critical
assessment in calculating power usage effectiveness for a data centre, Energy
Convers. Manage. 76 (2013) 155–161.
[22] ISO/IEC Guide 98-3:2008, Uncertainty of measurement – Part 3: Guide to the ex-
pression of uncertainty in measurement (GUM:1995).