Test Results from the Comparison of Three Liquid Cooling Methods for High-Power Processors

Guy R. Wagner, Joseph R. Schaadt, Justin Dixon, Gary Chan, William Maltz – Electronic Cooling Solutions
Kamal Mostafavi - Coolit Systems
David Copeland – Oracle

Electronic Cooling Solutions Inc.

2915 Copper Road
Santa Clara CA 95051
Email: gwagner@ecooling.com

ABSTRACT KEY WORDS: Processor Cooling, Single-Phase, Two-Phase,

This study compares three different liquid cooling
technologies to determine which of the three methods is able
to cool the highest power density processor chips.
The first method consists of pumping a liquid coolant
through a cold plate mounted over a 25.4 mm square heat
source. The second method is two-phase immersion cooling
of the 25.4 mm square heat source in a bath of 3M Novec ®
649 liquid with a boiling point of 49°C. The third method of
cooling consists of single-phase immersion cooling of the 25.4
mm square heat source using mineral oil as the coolant. This
method was tested under both natural convection and forced
convection conditions.
An experimental setup, consisting of a heater module with
attachment to the cooling solution was designed and built. The
apparatus was used to determine the maximum power
dissipation that can be supported in a 25.4 mm x 25.4 mm area
for each of the three cooling solutions. Note that uniform
power dissipation is assumed over the die surface. The
measured temperature is assumed to be equivalent to the
temperature of a die with the same surface area. The
maximum die temperature was set to 100 °C and the power
required to reach that temperature was recorded for each of the
cooling methods.
The maximum power dissipation achieved and the thermal
resistances for the three solutions are shown below:
Single-phase cold plate – 1,480 watts
Thermal resistance: 0.048 K/W
2-Phase immersion – 740 watts
Thermal resistance: 0.056 K/W
Single-phase mineral oil immersion – 450 watts
Thermal resistance: 0.140 K/W
Based on the results from testing the constructed setups,
the single-phase cold plate provided by CoolIT Systems was
able to achieve the highest power dissipation with the lowest
thermal resistance. The 2-phase immersion cooling using 3M
Novec 649 performed second best and the single-phase
mineral oil immersion solution had the lowest performance.
Each cooling technology has advantages and
disadvantages. In some instances, power for pumps was
required to circulate fluid. Each of the three setups used
facility water to remove heat from the system.
Immersion Cooling, Cold Plate, Impingement Cooling, Novec
649, Direct-to-Chip Cooling, Forced Convection, Natural
Convection
INTRODUCTION
This study aims to evaluate three different liquid cooling
solutions and create a comparison of each technology’s ability
to cool high power density processor chips. The study was
prompted by the need to cool extremely high power processors
such as used in Bitcoin mining [1][2].
The first liquid cooling method to be examined is using a
single-phase cold plate provided by CoolIT Systems Inc.
which is placed directly over a pseudo computer chip. In this
study the pseudo computer chip is a 25.4 mm square copper
block attached to a ceramic heat source of the same size. The
second method investigated is two-phase immersion cooling
using 3M Novec 649 as the coolant on the pseudo computer
chip. The third method of cooling investigated in this study
uses single-phase immersion cooling in mineral oil to cool the
pseudo computer chip through both natural convection and
forced convection [3][4]. This study presents experimental
results of the three liquid cooling methods and the
experimental setups designed to examine the thermal
effectiveness of each method.
EXPERIMENTAL DESIGN
HEATING MODULE (PSEUDO COMPUTER CHIP)
This study used a ceramic resistive heat source mounted to
a copper block to simulate the thermal pattern of a high
power-density processor chip. The dimensions of the heat
source are 25.4 mm by 25.4 mm square with a height of 2.3
mm. Attached to the bottom of the ceramic heater is a 50 mm
by 50 mm by 15.3 mm insulator with a thermal conductivity
of 0.2 W/m-K. A 25.4 mm by 25.4 mm by 5.0 mm copper
block acting as the die is attached directly to the top of the
ceramic heater. All three of the cooling solutions were
attached to the copper die surface. From simulations, it was
determined that over 99% of the dissipated power from the
heater flows into the copper block serving as the die. The fill
material and polycarbonate base provide an insulating medium
to keep the heat flow very low from the sides and bottom of
the heater and sides of the copper block. Figures 1 and 2
below show the heater base in the preliminary stage and when
a cooling solution, in this case a heat sink, is mounted.
The ceramic heater, supplied by Component General, has a
maximum temperature of 150°C. Power is limited in the

978-1-4673-8121-5/$31.00 ©2016 IEEE 619 15th IEEE ITHERM Conference

heater module such that the thermocouple embedded in the SINGLE-PHASE COLD PLATE COOLING SOLUTION
center of the copper block does not have readings that exceeds
The liquid cooling loop design employed a single-phase
100°C. This is done so the maximum temperature limit of the
microchannel cold plate solution. The loop contained a liquid
heater block is not exceeded and because the case temperature
to liquid heat exchanger, a Laing Thermotech liquid pump, a
limit of a high performance chip is typically below 100°C.
Rotameter flow meter, and a CoolIT R3 microchannel cold
plate. The liquid used in the loop was CoolIT A2 coolant and
the loop was instrumented with thermocouples at the inlet to
the cold plate, in the center of the copper block, the surface of
the cold plate, and at the outlet of the cold plate. This can be
seen in Figure 4.

Fig. 1 Common Heater Base

Fig. 4 Single-Phase Cold Plate Cooling Solution Setup

The A2 fluid used as the coolant has the following properties:

0.51 W/mK thermal conductivity
0.001665 Ns/m3 viscosity
4031 J/kg-K specific heat
1014 kg/m3 density

The cold plate used for the experiments is shown in figure 5.

Fig. 2 Cross-Section View of the Common Heater Base

The high power being dissipated in the 6Ω ceramic heater

required an electrical circuit to be designed for the heater. The
voltage across the heater and the current through the heater
were both measured and monitored during the experiment to
ensure consistent values. The air-cooled 0.05Ω shunt resistor
used for measuring the current was chosen so that it could
handle the 20W power dissipation. Three power supplies
where used since the largest power supply in the inventory
could not supply the required voltage and current to achieve
the required power dissipation. The output of the power
supplies was manually adjusted during the experiments.

Fig. 5 Cold plate used for the single-phase experiments

SINGLE-PHASE OIL IMMERSION COOLING

The single-phase oil immersion solution was designed with
a liquid to liquid heat exchanger, two liquid pumps, and an
impingement nozzle manifold. One pump was used to deliver
the mineral oil coolant to the manifold while the other pump
Fig. 3 Electrical Circuit Diagram for 6Ω Heater
was used to circulate the mineral oil through the liquid-liquid
heat exchanger. Figure 6 shows the setup.

Fig. 6 Single-Phase Oil Immersion Setup

The nozzle manifold was designed such that the mineral oil
impinged equally over the pin-fin copper heat sink providing
for uniform convective heat transfer. The impingement device
can be seen below in Figure 7.
Fig. 8 Two-Phase Testing Apparatus
The heater base was then placed in the center of a fluid tank.
A vapor condenser coil was placed at the top of the fluid tank
as shown in Figure 9. The heater module and heat sink were
then submerged in NovecTM 649. The vapor condenser, which
is made of a ¼” copper tube, is positioned 3” above the liquid
level. Facility water runs through the copper tube to remove
heat from the system through condensation of the vapor [8][9].

Condenser
Coil

Fig. 7 Nozzle Manifold

The immersion cooling experiment used a standard copper
pin-fin heat sink. For the natural convection testing the nozzle
manifold was removed, the common heater base was Fluid Tank
positioned both vertically and horizontally for separate natural
convection cases. The natural convection cases only used the
circulation pump to cool the oil bath.
Fig. 9 Two-Phase Setup
The mineral oil used as the coolant has the following
properties: The fluid selection decision for the two-phase immersion
0.13 W/mK thermal conductivity cooling setup was based on the criteria of boiling point at
0.0379 Ns/m3 viscosity atmospheric pressure, latent heat, global warming potential,
1927 J/kg-K specific heat and the di-electric constant. A visit was made to the
850 kg/m3 density headquarters of 3M in St. Paul, Minnesota to discuss the fluid
selection with Phil Tuma. After inspecting a NovecTM 649
setup at the 3M facility, a decision was made to use NovecTM
TWO-PHASE IMMERSION COOLING 649 in the two-phase immersion experiments because of its
low boiling point, low dielectric constant and low global
The two-phase immersion cooling solution used the common warming potential.
heater base with a pin-fin heat spreader on top of the copper
block. The pin-fin heat spreader was coated with a boiling
enhancement coating (3M L-20227) that enhances boiling heat
transfer [5][6][7]. The setup with the pin-fin heat spreader can
be seen in Figure 8.
 Fluorinert™ Perfluorocarbon
3M Tradename(PFC)and Chemistry Fluorinert™
Novec™Perfluorocarbon
Hydrofluoroether(PFC)
(HFE) Novec™
Novec™ Fluoroketone
Hydrofluoroether
(FK)(HFE)
284 Product
FC-72 Name FC-84 Novec™
FC-3284 7000 Novec™
FC-72 7100 FC-84
Novec™ 7200 Novec™ 7000 Novec™ 649 Novec™ 7100 Novec™ 774 Novec™ 7200
NO Molecular
C6F14 Formula C7F16 CC3F57
FOCH
11NO 3 C
C64FF14
9OCH3 C7FC 164F9OC2H5 C3F7C OCH
2F5C(O)CF(CF
3 3)2C4F9COCH
3F7C(O)CF(CF
3 3)2C4F9OC2H5
9 Molecular
338 Weight [g/mol]
388 200
299 338250 388 264 200 316 250 366 264
several
Number Isomers several 11 several 2 several 2 1 1 2 2 2
Normal
56 Boiling Point [°C]
80 3449 56 61 80 76 34 49 61 74 76
1 Critical
176 Temperature 202 [°C] 165.0
161 176195.2 202 209.8 165.0 169 195.2 195 209.8
7 Critical
1.83 Pressure [MPa] 1.74 2.48
1.87 1.83 2.23 1.74 2.01 2.48 1.87 2.23 1.71 2.01
Vapor
30 Pressure at 25°C 11 [kPa] 6535 30 27 11 16 65 40 27 16 16
0 Surface
10.0 Tension [dynes/cm]
12 12.4
13.0 10.0 13.6 12 13.6 12.4 11.4 13.6 12.3 13.6
3
0 Liquid
1680 Density [kg/m1730] 1401
1610 16801510 1730 1420 1401 1610 1510 1.67 1420
0 Specific
1100 Heat [J/kg-K] 1100 1300
1100 11001180 1100 1220 1300 1103 1180 1130 1220
2 Thermal
0.057 Conductivity0.06 [W/m-K] 0.075
0.062 0.057
0.069 0.06 0.068 0.075 0.059 0.069 0.06 0.068
5 Latent
88 Heat [kJ/kg] 90 142
105 88112 90 119 142 88 112 90 119
Solubility
10 H2O in Fluid 11[ppmw] 6014 10 95 11 92 60 20 95 1.8 92
Dielectric
1.8 Constant at1.8 1kHz 7.4
1.9 1.87.4 1.8 7.3 7.4 1.8 7.4 1.8 7.3
00 Global >9000 1
>9000 Warming Potential >9000
370 >9000 280 >9000 55 370 1 280 1 55
8-Hour
NA Exposure Limit NA[ppmv] 75
NA NA750 NA 200 75 150 750 ND (~150) 200
Acute
NA Lethal Conc. [ppmv]
NA >30,000
NA >100,000
NA NA >92,000 >30,000>100,000 >100,000 <100,000 >92,000
Cardiac
N Sensitizer N NN N N N N N N N N N
ertness, low tox,
Advantage
compat with alumina high
extreme
latent heat,
inertness,
low GWP
low tox, compat with alumina high latent
lowest
heat,GWP,
low GWP
PFC-like dielectric
, low latent heat
Disadvantage higher
highhydrocarbon
GWP, low latent solvency,
heat reduced dielectric props higher hydrocarbon
incompatible solvency,
with liquid
reduced
water dielectric props
Table 1 Properties of 3M NovecTM cooling fluids

EXPERIMENTAL RESULTS
SINGLE-PHASE COLD PLATE COOLING SOLUTION

The measured temperatures were found to be a linear

relationship over the 0 to 1,440 Watt power range. The
coolant inlet temperature shows a slight increase at elevated
power dissipation values as shown in Figure 10.

Fig. 11 Measured Temperature Relative to Coolant Inlet

Temperature

The thermal resistance was found by dividing the

temperature difference between the copper block and the inlet
coolant temperature by the power. Figure 12 shows the
thermal resistance of the copper block as a function of the
coolant flow rate. The thermal resistance was found to be
0.048 K/W at 800 Watts and 0.050 K/W at 300 Watts with a
1.39 liters/min coolant flow.
Fig. 10 Measured Temperatures vs. Power for Cold Plate

The outlet coolant temperature increased by 15 °C relative

to the inlet coolant temperature at a 1,400 Watt power
dissipation. The measured temperature increase of the copper
block (die) relative to the coolant inlet temperature is plotted
in figure 11.

Fig. 12 Thermal Resistance vs. Flow Rate

SINGLE-PHASE OIL IMMERSION COOLING
Using natural convection, the maximum power dissipation is
140 Watts for the horizontal orientation and 160 Watts for the
vertical orientation. The temperature rise of the copper block
was 60 °C for an average oil bath temperature of 40 °C.
Figure 13 shows the relative temperatures for horizontal and
vertical orientations with respect to power.

Fig. 15 Thermal Resistance vs. Power

TWO-PHASE IMMERSION COOLING

The copper block temperature was found to have a fairly

linear relationship with power over the 0 to 750 Watt range as
shown in figure 16.

Fig. 13 Relative Temperatures vs. Power

For forced convection, the setup was able to dissipate up to
400 Watts with a 10.6 mm gap between the nozzle manifold
and the heat sink and 450 watts when that gap was decreased
to 2.6 mm. Fig. 14 shows the copper block temperature
plotted with respect to power.

Fig. 16 Copper Block Temperature vs. Power

The two-phase flow was observed clearly in the fluid bath as

shown in Figure 17.

Fig. 14 Tblock vs. Power

The thermal resistance was calculated by taking the difference
between the copper block temperature and the oil temperature
and dividing it by the power. The optimal thermal resistances Fig. 17 Two–Phase Flow Behavior in a Fluid Bath
are 0.14 K/W and 0.16 K/W for impingement setups and 0.37
K/W and 0.41 K/W for setups with natural convection as The copper block temperature was plotted with respect to time
shown in figure 15. in Figure 18.
Fig. 18 Temperature Response to a Power Change
The optimal thermal resistances for the two-phase setup were
0.056 K/W for the horizontal orientations and 0.067 K/W for
the vertical orientation. As surface contact between the fluid
and heat sink begins to decrease because of an increase in
vapor pockets, the thermal resistance was observed to
increase. Notice that, unlike the single phase natural
convection immersion cooling, the two phase cooling
performs best in the horizontal position. The reason is that the
fluid in contact with the heat sink can be replenished faster if
the vapor bubbles can flow rapidly away from the surface
being cooled. In the vertical position, the vapor generated by
the lower section of the heat sink stays next to the upper
surface and impedes the fluid flow to that surface.
SUMMARY & CONCLUSIONS
The CoolIT cold plate solution performed the best of the
three liquid cooling solutions evaluated. The cold plate was
able to properly dissipate 1,480 Watts before reaching the
100C copper block limit. The two-phase immersion cooling
using 3M NovecTM 649 was able to dissipate 740 Watts. The
single-phase mineral oil impingement immersion cooling was
able to dissipate 450 Watts. Note that the two-phase
immersion cooling solution may yield the highest packing
density in terms power dissipation per unit volume in the case
of a multi-processor system because it does require as much
volume for the hardware as the other cooling solutions.
The highest thermal resistance value was found with the
single-phase mineral oil in natural convection while the cold
plate solution had the lowest thermal resistance. This can be
seen in figure 19.
The maximum power dissipation for each of the three
cooling methods in this investigation was determined by the
temperature of the copper block. This was held to a maximum
of 100C to prevent damage to the ceramic heater.
Fig. 19 Thermal Resistance Comparisons
In conclusion, the CoolIT cold plate solution was found to be
the most effective in handling single processor chips with high
power densities.
ACKNOWLEDGMENTS
This study would like to acknowledge Phil Tuma from 3M
Corporation for his critical input on the two-phase fluid
selection.
Novec® is a registered trademark of 3M Corporation.
REFERENCES
[1] Kampl, A.,”Bitcoin 2-Phase Immersion Cooling and the
Implications for High Performance Computing”,
Electronics Cooling, March 2014, pp. 24-29.
[2] Taylor, M.B., “Bitcoin and the Age of Bespoken Silicon”,
2013 International Conference on Compilers, Architecture
and Synthesis forEmbedded Systems (CASES). Sept 29 –
Oct. 4, 2013
[3] Tulkoff, C., “Improved Efficiency & Reliability for Data
Centers Using Immersion Oil Cooling”, IMAPS 2012
Advanced Technology Workshop on Thermal
Management, Los Gatos, CA., Nov 12-14 2012
[4] Merritt, R., “Intel Tests Oil Immersion to Cool Servers”,
EE Times, 9-5-2012
[5] Campbell, L. and Tuma, P., “Numerical Prediction of the
Junction-to-Fluid Thermal Resistance of a 2-Phase
Immersion-Cooled IBM Dual Core POWER6 Processor”,
Proc. 28th IEEE SEMI-THERM Symposium, San Jose,
CA, March 2012, pp. 36-44.
[6] US Patent 7,360,581
[7] US Patent 7,695,808
[8] Tuma, P., “Design, Fabrication and Operation of a 1.2kW
Open Bath Immersion Cooled Server Demo”, IMAPS
2012 Advanced Technology Workshop on Thermal
Management, Los Gatos, CA., Nov 12-14 2012
[9] Tuma, P.E., “The Merits of Open Bath Immersion Cooling
of Datacom Equipment”, Proc. 26th IEEE SEMI-THERM
Symposium, San Jose, CA, March 2010, Session 5