See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/337848891

Computational Form Factor Study of a 3rd Generation Open Compute Server for
Single-Phase Immersion Cooling

Conference Paper · October 2019

DOI: 10.1115/IPACK2019-6602

CITATIONS READS

10 678

6 authors, including:

Jimil M. Shah Satyam Saini

TMGcore Intel
39 PUBLICATIONS 323 CITATIONS 35 PUBLICATIONS 205 CITATIONS

SEE PROFILE SEE PROFILE

Dereje Agonafer
University of Texas at Arlington
290 PUBLICATIONS 1,791 CITATIONS

SEE PROFILE

All content following this page was uploaded by Jimil M. Shah on 11 December 2019.

The user has requested enhancement of the downloaded file.

 A
Proceedings of the ASME 2019 International Technical Conference and Exhibition

tte
on Packaging and Integration of Electronic and Photonic Microsystems

nd
ee
InterPACK2019
October 7-9, 2019, Anaheim, CA, USA

R
ea
d-
O
nl
IPACK2019-6602

y
C
op
y
COMPUTATIONAL FORM FACTOR STUDY OF A 3RD GENERATION OPEN COMPUTE
SERVER FOR SINGLE-PHASE IMMERSION COOLING
1
Jimil M. Shah , Chinmay Bhatt, Pranavi Rachamreddy, Ravya Dandamudi, Satyam Saini and
Dereje Agonafer
University of Texas at Arlington
Arlington, Texas, USA
ABSTRACT lpm liter per minute
Networking and computing dependency has been μ Dynamic Viscosity
increasing in the modern world, thus, boosting the growth of ν Kinematic Viscosity
data centers in leading business domains like banking, Pr Prandtl Number
education, transportation, social media etc. Data center is a 𝑅𝑒 Reynolds Number
facility that incorporates an organization's IT operations and V Flow Velocity
equipment, as well as where it stores, processes and manages W Watt
the data. To fulfill the increasing demands of data storage and
data processing, a corresponding increase in server 1. INTRODUCTION
performance is needed. This causes a subsequent increment in Data Center is a part of the industry, which facilitates
power consumption and heat generation in the servers due to storage, processing and managing of important data by housing
high performance processing units. Currently, air cooling is the servers mounted in racks. In today’s networking world,
most widely used thermal management technique in data utilization of servers and data centers has been increasing
centers, but it has started to reach its limitations in cooling significantly. Increasing demands of processing and storage of
high packaging densities. Therefore, industries are looking for data cause a corresponding increase in power density of
single-phase immersion cooling using various dielectric fluids servers. Computer system dependency has been increased and
to reduce the operational and cooling costs by enhancing the that has encouraged the rapid growth of data centers in leading
thermal management of servers. This research work aims at business units like banking, education, transportation, social
increasing the rack density by reducing the form factor of a 3 rd media etc. Data center unit is one of the leading power
Generation Open Compute Server using single-phase consumption sectors of an industry [1-9].
immersion cooling. A computational study is conducted in the Increased demand of data centers requires efficient cooling
operational range of temperatures and the thermal efficiency is technology. Increase in the power density of servers directly
optimized. A parametric study is conducted by changing the increases the amount of heat generation. To ensure the
inlet velocities and inlet temperatures of cooling liquid for reliability and efficient working characteristics of the server
rd
different heights of the open compute 3 generation server. A components, servers must be cooled continuously and
comparative study is then carried out for white mineral oil and efficiently. The data center energy efficiency largely depends on
synthetic fluid (EC100). thermal management of servers [1-9].

Keywords: open compute servers, single-phase Data Center Cooling Methods

immersion cooling, form factor, thermal mass At present, there are various methods of data center
cooling used for maintaining data center facility under
NOMENCLATURE permissible temperature range. Two major cooling techniques
b channel width are:
Cp Specific heat capacity
CRAC Computer Room Air Conditioning 1. Air cooled servers
CPU Central Processing Unit 2. Liquid cooled servers
D Hydraulic Diameter
DLC Direct Liquid Cooling Air Cooled Servers
DIMMs Dual In-line Memory Module Air cooled configuration is the most widely used cooling
h Channel height technique for data center applications. Air cooled servers
IT Information Technology work on forced convection of air over a heat transferring
component called a heat sink. The Cooling system of an air-
1
Contact author: jimil.shah@mavs.uta.edu cooled server consists of axial fans to handle the air volume

1 Copyright © 2019 ASME

and heat sinks to increase the heat transfer. Basically, heat
sinks are the components that transfer the heat generated by
the processor to ambient air through three modes of heat
Heat
transfer, i.e. conduction, convection and radiation. Fans are
Type of Density Kinematic Heat
used to manage the airflow that can carry the heat from the Capacity
server component from inlet to outlet. The fans are controlled Fluid (Kg/m3 ) viscosity Conductivity
as per the change in the temperature of the server component. (KJ/Kg
Exhaust hot air from the server outlet enters the hot aisle and (X (W/m K)
is then directed to the Computer Room Air Conditioning K)
(CRAC) unit to cool the air up to the required temperature. 10−6 m2 /s)
That cooled air is be supplied back to the cold aisle of the
Air 1.01 1.225 0.016 0.02
data center [3].

Liquid Cooled Servers Water 4.19 1000 0.66 0.58

Currently, air cooling is the most widely used thermal
management technology in data centers. However, air cooling
has started reaching its limits due to high powered White 1.67 849.3 16.02 0.13
microprocessors and dense packaging [7]. Moreover, air cooled
Mineral
servers consume more space as they must house the ducting
system and fans. To overcome these limitations of air cooling Oil
in data centers, liquid cooling methods have started gaining
more attraction. Liquid cooled servers are subdivided into two
categories depending upon the cooling fluid used. Synthetic 2.165 803.78 13.22 0.1378

a. Water cooled servers fluid (EC-

b. Immersion cooled servers
Mineral oil is a dielectric fluid and it is being used since
decades in transformers [1-9]. The next radical change in the
thermal management of data centers is to shift from
conventional cooling methods like air-cooling to direct liquid
cooling (DLC) to deal with high thermal mass. Passive single-
phase dielectric fluid immersion cooling has several other
benefits like better server performance, even temperature
profile and higher rack densities [6-9]. In this study, we have
used white mineral oil and synthetic fluid Electro Cool 100
(EC-100). The physical properties of different types of fluids
are mentioned below in Table 1.
It is clear from the Table1 that the Thermal mass (Density
x Heat Capacity) of dielectric liquids is high compared to air.
Although the thermal mass of water is the highest, water is a
conductive fluid and servers cannot be immersed in water
directly.
100)
Table 1: Properties of different fluids
Rack unit and Open Rack Unit
A rack unit is a unit of measurement of the overall height
of servers, server racks and other equipment that these racks
house. One U is equal to 1.75 inches or 44.45mm and a
standard rack frame measure is 42U. The equipment that
goes in these racks is generally manufactured in the multiples
of rack units. The open rack unit is slightly taller than a rack
unit and is generally 48mm tall. This added height has
various benefits like increased airflow and better cable
management. Open rack is a mounting system initiated by
Facebook’s Open Compute Project designed for better space
efficiency than the typical 19 inch racks. It has 21inches out
of the 24inches column width available for equipment,
yielding 87.5% space efficiency. Similarly, Table 2 provides
the values of heights of the chassis for different rack units
and open rack units [10].

UNIT RACK UNIT OPEN RACK UNIT

1U 44.45 48

1.5U 66.5 72

2U 89 96

2 Copyright © 2019 ASME

Table 2: Heights of the chassis for rack unit and open rack unit

The main purpose of this study is to get an insight into

increasing the existing rack density by reducing its form factor
for immersion cooling. This study aims at improving the rack
density by reducing the form factor of a 3rd generation Open
Compute Server for single-phase immersion cooling using
computational analysis. The analysis is validated for two
dielectric fluids: white mineral oil and synthetic fluid, EC100.
The CFD study is carried out for a varying range of operational
inlet temperatures. Inlet temperatures and flow rates were
parameterized for 1U, 1.5U and 2U Open Rack Unit form
factor values to optimize thermal efficiency.

2. MODELING

2.1 Server Description

The server taken under study is the Third Generation Open

Compute Server. It comprises of two microprocessors having a FIGURE 1: CFD MODEL OF A BASELINE OIL COOLED
design power density of 115 W each and dimensions of 50mm SERVER
X 50mm. As shown in Figure 1, the server houses four DIMM 2.2 Velocity Calculations:
blocks, each having four DIMMS of 8GB memory each. Intel The velocity of the fluid changes whenever the volumetric
based server has a form factor 2U open rack unit. It has flow rate and cross-sectional area of the domain changes. Table
dimensions of 166.2mm x 511mm x 96mm (l x b x h). The 3 below shows the velocity calculation for various open rack
server contains two fans as it is originally designed for air unit heights with varying flow rate using the equations given
cooling. The fan has the dimensions as 80mm×80mm×38mm. below.
The speed of the fan is controlled by the change in CPU die
temperature. Heat sinks are mounted on the CPU to transfer the Volume flow rate = Area x Velocity
heat from processors to air.
Baseline oil cooled server has been developed using a
computational tool, Ansys Icepak as shown in Figure 1. The Velocity = Volume flow rate/Area
fans and ducting system were removed. Server dimensions are
kept similar as an air-cooled server except for the height of the
ceiling is kept as two fin space above the height of the plate fin
heat sink. LPM 1 ORU 1.5 ORU 2 ORU

(m/s) (m/s) (m/s)

0.5 0.0010442 0.00069639 0.00052209

1 0.0020896 0.0013928 0.0010448

1.5 0.0031337 0.002089 0.00156689

2 0.00417795 0.002785 0.0020889

2.5 0.005223 0.003482 0.0026115

Table 3: Velocity Calculations

There are different types of dielectric fluids, but in this

study, the properties of White Mineral Oil and Synthetic Fluid
EC100 are used. 2 Open Rack Unit Server cross section in this
case is considered as 511mm x 166.2mm x 96mm. The formula
below can calculate Hydraulic Diameter,

3 Copyright © 2019 ASME

 2𝑏ℎ The properties that must be considered for the calculation
𝐷= (1) of Reynolds number for synthetic fluid, EC 100 are as follows:
(𝑏 + ℎ)
Where, Density – 803.78 kg/m3
b= 511 mm Thermal conductivity – 0.1378 W/m K
h= 96 mm Specific heat – 2165.9 J/kg k
D = (2 x 511 x 96) / (511 +96) Molecular Weight ̶ 350 kg/ K Mol
= 161.6 mm
= 0.1616 m At 30℃, the Dynamic Viscosity (μ) of Synthetic Fluid is
0.01062 (kg/m s). The Kinematic Viscosity (ν) is 1.322E-05
2.3 Reynolds number calculation for White Mineral Oil m2 /s. The Reynolds number therefore is calculated as given
below:
These are some physical properties of white mineral oil, VD
𝑅𝑒 =
considered for computational analysis [11]: ν
Density – 851.515 kg/m3 Re = (0.0010448 x 0.1616) / 1.322E-05
Thermal conductivity – 0.13 W/m K Re = 12.77
Specific heat – 1680 J/kg k Prandtl number can be calculated using the below formula,
μC
Thermal Diffusivity – 9.166E-8 m2 /s Pr= P = (0.01062 X 2165.9) / 0.13787
λ
Molecular Weight ̶ 150 kg/ K Mol Pr= 166.837
Overall heat transfer co-efficient – 50-30 W/ m2 𝐾 Similarly, the values of Re and Pr are calculated for other
temperature values and are shown in Table 5.
At 30℃, the Dynamic Viscosity (μ) of White mineral oil is
0.01405 kg/(m s). The Kinematic Viscosity (ν) is 1.65E-05 Temperature Dynamic Kinematic Reynolds Prandtl
m2 /s. It is very important to make the decision of the solver (℃) Viscosity Viscosity Number Number
model. To decide that, Reynolds number must be calculated. (kg/ms) (m2/s)
Reynolds number can be calculated using the below formula:
VD 30 0.01062 1.32E-05 12.77 166.84
𝑅𝑒 = (2)
ν 40 0.00767 9.63E-06 17.53 123.08
Where Re is the Reynolds number, V is the Velocity, D is the
Hydraulic Diameter, and ν is the Kinematic Viscosity. 45 0.00662 8.34E-06 20.24 107.35
Substituting Hydraulic Diameter, Velocity and Kinematic
Viscosity in eq 2, 50 0.00576 7.29E-06 23.16 94.39
Re = (0.0010448 x 0.1616) / (1.65E-05) Table 5: Temperature dependent properties of Synthetic Fluid
Re = 10.23
Prandtl number can be calculated using the below formula: 2.5 Parametric study of form factor
μC
Pr= P (3)
λ
= (0.01405 X 1670) / 0.13 The server is basically designed for air cooling application
Pr= 180.488 and it has by default form factor as 2U open rack unit. The
Similarly, the values of Re and Pr are calculated for other baseline oil-cooled server has been developed using the
temperature values and are shown in Table 4. computational tool Ansys Icepak. Server dimensions were kept
the same as the baseline server except the duct and fans were
Temperature Dynamic Kinematic Reynolds Prandtl removed. In this geometry, the cabinet maximum y dimension
Viscosity Viscosity Number Number depicts the inlet and minimum y depicts the outlet for the flow
(℃)
(kg/ms) (m2/s) of fluid. Ansys Icepak has the provision to change the
parameters and solve the Navier-Stokes equations. Two major
30 0.01405 1.65E-05 10.23 180.49 parameters have been considered are inlet oil temperature and
40 0.01046 1.23E-05 13.72 134.37 oil flow rate. Inlet oil temperature has been varied from 30℃ to
50℃. Volume flow rate range for an incoming oil is kept from
45 0..00909 1.07E-05 15.77 116.77 0.5 lpm to 2.5 lpm. Reynolds number does not exceed 2000
50 0.00794 9.35E-06 18.05 101.99 hence, laminar model is used for solution of parametric trails.
Similar procedure repeated for form factors 1.5U open rack unit
Table 4: Temperature dependent properties of a White Mineral and 1U open rack unit. The Cabinet height is changed
Oil according to the form factor value. This study is very useful to
predict the behavior of cooling fluid at various form factors.
2.4 Reynolds Number calculation for EC-100 Small improvement at server level cooling can be useful for a
significant amount of savings at the facility level [12].

4 Copyright © 2019 ASME

3. RESULTS AND DISCUSSION

FIGURE 2: TEMPERATURE CONTOURS ON BOTH THE CPUS

WITH MINERAL OIL
A series of simulations were performed by changing the
properties depending on the temperature and velocities when
the height of the server is changed. The purpose of this study is
to analyze the change in maximum junction temperature and
thermal resistance when form factor of the server is reduced.
Figure 2 shows the temperature contours on both the CPUs
FIGURE 3: MAX JUNCTION TEMP VS OIL FLOW RATE AT
with mineral oil as the cooling fluid. The thermal properties are SOURCE 2 FOR 2U OPEN RACK UNIT
changed depending on the fluid used and the inlet conditions.
Form Factor study of Facebook’s third generation open
compute server includes comparison of thermal resistances and
maximum junction temperatures of 2U Open Rack Unit, 1.5U
Open Rack Unit and 1U Open Rack Unit for low velocity oil
cooled severs as shown in figures 3-8. The results show the
change in a trend of thermal performance of the server with
variation in inlet boundary conditions.

5 Copyright © 2019 ASME

FIGURE 4: THERMAL RESISTANCE VS OIL FLOW RATE AT FIGURE 5: MAX JUNCTION TEMP VS OIL FLOW RATE AT
SOURCE 2 FOR 2U OPEN RACK UNIT SOURCE 2 FOR 1.5U OPEN RACK UNIT

FIGURE 6: THERMAL RESISTANCE VS OIL FLOW RATE AT

SOURCE 2 FOR 1.5U OPEN RACK UNIT

6 Copyright © 2019 ASME


FIGURE 7: MAX JUNCTION TEMP VS OIL FLOW RATE AT
SOURCE 2 FOR 1U OPEN RACK UNIT
FIGURE 8: THERMAL RESISTANCE VS OIL FLOW RATE AT
SOURCE 2 FOR 1U OPEN RACK UNIT
7 Copyright © 2019 ASME
It can be seen in Figure 3 and Figure 4 that the maximum
junction temperature at 50°C is between 70-80°C which is
below the thermal design limits of the processor. A similar trend
is observed for 1U and 1.5U Open Rack unit servers as shown
in Figures 5, 6, 7 and 8. It is also concluded from Figure 4 that
at same flow rate and inlet temperature, EC100 performs
slightly better in keeping the processor temperature lower as
compared to white mineral oil. Of all the simulations run on the
server for different inlet temperatures, we obtained the best
results for 1.5 Open rack unit server at 40℃ and a flow rate of 1
lpm using synthetic fluid. It is thus advisable to keep the oil
flow rate in the range of 1 to 2.5 lpm for effective cooling of
servers, especially with 1.5 Open rack unit server form factor.
From the Figure 5, it is evident that even at 40℃ inlet
temperature; the maximum junction temperature at the
CPU is well below the design limits.
4. CONCLUSION
FIGURE 9: REDUCING SERVER FORM FACTOR
It can be well concluded from this study that single-phase
immersion cooling can keep high powered IT equipment
temperature within the recommended temperature limits.
Another advantage of the single-phase immersion cooling lies
in the fact that the server space can be optimized (Figure 9) as
there is no need for ducting as in the case of air-cooled servers.
The comparative study also shows an enhancement in thermal
management when the synthetic fluid is used but not a
significant difference. For the future study, Fluorochemical
fluids should be considered as Figure of Merit is higher for
such fluids than FOM of Hydrocarbon fluids. Also, as it can be
noted, the flow rates required for cooling are quite low which is
generally less than the power consumed by fans in air-cooled
servers. Thus, submerging servers and other IT equipment in
dielectric fluids can be viewed as a viable option for substantial
cost savings in terms of cooling energy expended. It also
eliminates the limitations of traditional air-cooling methods in
terms of capability to cool higher power densities. The server
architecture becomes less complex with improved performance.
ACKNOWLEDGEMENTS
This work is supported by NSF I/UCRC in Energy-Smart
Electronic Systems (ES2).
oil” by Prof. B.Pahlavanpour, Dr. M. Eklund, Mr. K.Sundkvist,
Nynas Naphthenicgy.
[12] Shah, J. M., 2018, “Characterizing Contamination To
Expand ASHRAE Envelope In Airside Economization And
Thermal And Reliability In Immersion Cooling Of Data
Centers”, PhD Dissertation, University of Texas at Arlington,
Arlington, TX.

REFERENCES
[1] Shah, J. M., Eiland, R., Rizvi, S. H. I., and Agonafer, D.,
2015, “Critical Non-Thermal Considerations for Oil Cooled
Data Centers,” IMAPS Advanced Technology Workshop and
Tabletop Exhibit on Thermal Management, Los Gatos, CA,
Sept. 22–24.
[2] Shah, J. M., and Agonafer, D., 2015, “Issue on Operational
Efficiency for Oil Immersion Cooled Data Centers,” ASME
International Mechanical Engineering Congress and
Exposition.
[3] Shah, J. M., Rizvi, S. H. I., Kota, I. S., Nagilla, S. R.,
Thakkar, D., and Agonafer, D., 2016, “Design Considerations
Relating to Non-Thermal Aspects of Oil Immersion Cooling,”
ASME Paper No. IMECE2016-67320.
[4] Eiland, R., Fernandes, J., Vallejo, M., Agonafer, D., and
Mulay, V., 2014, “Flow Rate and Inlet Temperature
Considerations for Direct Immersion of a Single Server in
Mineral Oil,” 14th Intersociety Conference on Thermal and
Thermomechanical Phenomena in Electronic Systems
(ITherm), Orlando, FL, May 27–30, pp. 706–714.
[5] Eiland, R., Edward Fernandes, J., Vallejo, M., Siddarth, A.,
Agonafer, D., and Mulay, V., 2017, “Thermal Performance and
Efficiency of a Mineral Oil Immersed Server Over Varied
Environmental Operating Conditions,” ASME J. Electron.
Packag., 139(4), p. 041005.
[6] Shah, J. M., 2016, “Reliability Challenges in Airside
Economization and Oil Immersion Cooling,” Master’s thesis,
University of Texas at Arlington, Arlington, TX.
[7] Eiland, R., 2015, “Thermo-Mechanical Design
Considerations at the Server and Rack Level to Achieve
Maximum Data Center Energy Efficiency,” Ph.D. dissertation,
University of Texas at Arlington, Arlington, TX.
[8] Shah, J. M., Eiland, R., Siddharth, A., and Agonafer, D.,
2016, “Effects of Mineral Oil Immersion Cooling on IT
Equipment Reliability and Reliability Enhancements to Data
Center Operations,” 15th IEEE Intersociety Conference on
Thermal and Thermomechanical Phenomena in Electronic
Systems (ITherm), Las Vegas, NV, May 31–June 3, pp. 316–
325.
[9] Shah JM, Eiland R, Rajmane P, Siddarth A, Agonafer D,
Mulay V., “Reliability Considerations for Oil Immersion-
Cooled Data Centers”, ASME. J. Electron. Packag.
2019;141(2):021007-021007-9. doi:10.1115/1.4042979.
[10] J. Ning, “Intel Server in Open Rack Hardware v0.3 (MB-
draco-genam-0.3)", 2013.
[11] "Material safety data sheet" by STE oil company, Inc.
“Revised IEC standard for maintenance of in-service insulating

8 Copyright © 2019 ASME

View publication stats