Coldplate Design for Optimal Thermal Performance of High-Speed Electronics
Anil Yuksel
IBM Corporation
11400 Burnet Rd.
Austin, TX 78758, USA
Chris Marroquin
IBM Corporation
2800 37th St. NW
Rochester, MN 55901, USA
ABSTRACT
Thermal management of high-speed electronics has become
more challenging due to dramatic increase of power density in
high-speed CPUs and GPUs and server size limitations in data
center for next generation exascale high-performance
computing. Water cooling has been shown to provide better
cooling performance compared to air cooling; thus, systematic
investigation of water cooling in coldplate designs have been
increasingly demanding. Integration of such thermal cooling
designs into system level thermal management also creates
some issues in general due to the limitations of both
hydrodynamics and system level component's arrangements in
the servers. For instance, electrical routing and interconnect
architecture in electronic packaging create challenges for
thermal design of high-end servers. In this paper, we investigate
the optimal thermal performance of coldplate design by
analyzing the internal thermal resistance and the pressure drop
within the coldplate that can optimally be designed for high-end
server design. We also show the importance of modeling the
barb and riser assembly used in coldplate design to fully
understand the flow characterization in the coldplate in which
turbulent and vortex flow affects the pressure drop significantly
in overall hydrodynamic characterization of the coldplate.
KEY WORDS: Thermal design of high-speed electronics,
Liquid Cooling, Coldplate Design
INTRODUCTION
Thermal cooling and design of high-speed electronics, such as
CPU and GPU, has become more challenging recently due to
evolving silicon technologies along with shrinking size of
complementary metal-oxide-semiconductor (CMOS)
integrated circuits [1, 2]. Such scaling results in dramatic
increase in leakage current and density of power packaging. In
HPC (High Performance Computing) and Cognitive/AI System
development, parallel processing capabilities of the
supercomputers are essential within large clusters of servers
which requires multi-core Central Processing Units (CPUs) and
multi Graphics Processing Units (GPUs) [3, 4]. Moreover,
978-1-7281-9764-7/$31.00 ©2020 IEEE
Authorized licensed use limited to: University of Glasgow. Downloaded on November 01,2020 at 16:25:39 UTC from IEEE Xplore. Restrictions apply.
Vic Mahaney
IBM Corporation
11400 Burnet Rd.
Austin, TX 78758, USA
integration of those with many other electronic components
such as memory, network I/O, storage and solid/state devices in
the servers create a high power density in the server
architecture. Also, in most recent CPU and GPU applications,
multiple heat sources which also have different power density
are designed on the same silicon die area to improve the latency
and lower the delay between the memory and the processor
resulting in much higher bandwidth. For instance, multiple die-
stacked and TSV based HBM (High Bandwidth Memory)
technology are used near the processing unit chip for the next
generation interconnect and packaging applications. These
result in thermal and mechanical design challenges. As such,
efficient thermal cooling technologies need to be investigated
for thermal management of high-speed electronics. Liquid
cooling has increasingly been developed and used for the
thermal management of high power and closely spaced
electronic components such as GPUs and CPUs, which enable
higher heat transport capabilities than conventional air cooling
[5, 6]. Moreover, liquid cooling provides flexibility on
optimization of energy efficient data center design, noise
minimization and decreasing space requirements. In particular,
liquid cooling using a coldplate with embedded microchannel
has been applied to cool high power electronics [7, 8]. Due to
warm inlet liquid requirement (> 40 °C) and continuously
increasing power density of electronics within a restricted
packaging volume density, optimum thermal design is essential
for next-generation high-end servers. Thus, heat transfer
enhancement, pressure drop and mechanical assembly
development need to be understood well. Liquid cooling using
coldplate with microchannels can cool those high-speed
electronic devices and provide high reliable and good thermal
performance. Detailed review of fluid flow and heat transfer
through microchannels have been discussed [9]. Thermal
performance of single-phase liquid cooling by investigating the
Nusselt number, friction factor and pressure losses were
analyzed with correlations for laminar and turbulent flows and
compared to experimental data [10, 11]. Fully developed
hydrodynamic and thermal boundary layers have been
60 19th IEEE ITHERM Conference
discussed and the effect of size of the microchannel heat sink
design on overall thermal performance have been investigated
[12, 13]. Also, pumping and flow rate requirements for
microchannel heat sink design were characterized [14].
However, there are very few studies on flexible coldplate
design that can cool multiple devices as well as tolerating height
difference between closely spaced high power electronic
components with high reliability. Thus, we investigate the
thermal performance of different flexible coldplate assemblies
for optimal system level thermal and packaging design in this
study.
WATER COOLING DESIGN OPTIONS
Air cooling has traditionally been applied to cool the servers.
Nevertheless, increasing complexity and the power values in
high power electronics such as CPU and GPU makes air cooling
less favorable due to its lower thermal cooling capacity. Water
cooling; on the other hand, allows for higher power density.
Thus, water cooling through coldplate assembly can be
designed to cool effectively high power components with
dramatic increase of the electronic power components. As the
percentage of heat dumped to water is also very important in
data center for overall energy efficiency, air cooling via fans or
blowers at much lower speeds than acoustic limit can be
implemented to cool the low power electronic components on
the board in addition to the water cooling design aiming to cool
high power electronic components. Preheat from the high
power electronic components create a challenge at the back
section of the server in general which affects the location of the
components in the board, overall design, etc.. for purely air
configuration consisting of 2 IBM POWER CPUs (300 W each)
and 6 GPUs (400 W each). Five different cases shown in Fig.
2 are analyzed using an analytical analysis of 1D resistive type
network to understand the CPU and GPU’s temperature. In all
cases, a fixed flowrate (~1 gpm), warm water inlet temperature
(40°C) and measured coldplate thermal resistance data are used.
Case 1 shows the serial combination of the CPUs and GPUs.
This case has the advantage of the highest flowrate to each
coldplate, but has the disadvantage of the greatest preheat to the
final GPU coldplate. It is observed that there is 8.3°C bulk
heating for the 1800 W of preheat upstream of the last GPU and
the last GPU reaches 78 °C. Case 2 illustrates the two parallel
path approach for each CPU and 2 GPUs. Around 8.3 °C bulk
heating is observed for the last GPU which reaches at 77.5 °C.
Case 5 shows the parallel combination of the CPUs and GPUs.
This case has the disadvantage of the lowest flowrate to each
coldplate, but has the advantage of no preheat to any coldplate.
Hence, it can also be mentioned that this is the opposite of case
1. This results in the lowest temperature of the last GPU of 77.3
°C which is thermally desired option.
CASE I CPU CPU GPU GPU GPU GPU
CPU GPU GPU
CASE II
CPU GPU GPU
GPU GPU

cooled servers. However, preheat is not a major concern for the

water cooling of high power components via coldplate which CASE III CPU CPU
provides more flexibility on the overall design and the location
of the high power electronics on the board. Fig. 1 represents a
typical 2U (~3.5 inch) high-end server with a total power of GPU GPU

~3kW that has 2 CPUs and 4 GPUs.

GPU

GPU

CASE IV CPU CPU

GPU

GPU

GPU

CPU

GPU

Fig. 1. Water Cooling Design

CASE V
GPU

Hose dimensions & integration, cable routing, electrical

wiring and board design create typically some challenges on CPU
water cooling design. Thus, various water cooling and coldplate
design options and thermal performance analysis is essential for GPU
the optimal thermal and packaging design. Fig 2. illustrates
some possible water flow routing possibilities for a server Fig. 2. Schematic of various water cooling design cases

Authorized licensed use limited to: University of Glasgow. Downloaded on November 01,2020 at 16:25:39 UTC from IEEE Xplore. Restrictions apply.
 However, due to low flowrate (0.167 gpm) per path, the total GPU coldplate has an advantage over the pressure drop
external resistance on the CPU and GPU coldplate assembly compared to serial flow path option. Moreover, thermal penalty
reaches the maximum value which is 0.093 °C/W. This could due to bulk heating is expected less on the last GPU by using
bring some additional thermal and flow related challenges and 2x flexible coldplate. However, cost is expected to be less
high thermal penalty for part-to-part coldplate variations. As favorable due to having more copper tubes and mechanical
such, very low flowrate is typically not desired for long-term assembly components. Hose connections and cable
reliability and optimum thermal performance. management might create some spacing issues in parallel flow
Table 1. Thermal performance of analyzed water cooling option than serial flow option. However, contacting of 4 GPUs
cases by single 4x flexible coldplate in serial flow option is more
Case Qupstream Flowrate Flowrate DTbulk to DTbulk Total Rext Tinlet Tlast GPU challenging than 2 GPUs in parallel flow option. In general,
of last to last to last last for last DTbulk (ºC/W) (ºC) (ºC) leaving as many mechanical components untouched in GPU
GPU GPU GPU GPU GPU (ºC)
(W) (gpm) (lpm) (ºC) (ºC) assemblies is desired for better serviceability and reliability.

I 1800 1 3.8 6.8 1.5 8.3 0.078 40 78

II 700 0.5 1.9 5.3 3 8.3 0.081 40 77.5
III 400 0.33 1.3 4.6 4.6 8.3 0.085 40 78.4
IV 600 0.25 0.9 2.3 6 8.3 0.089 40 77.5
V 0 0.167 0.6 0 9.1 8.3 0.093 40 77.3

In summary, due to bulk heating and flowrate variations on

different five analyzed cases, the maximum temperature
difference is observed around 1.1 °C which is not significant.
Thus, further optimal design in CFD modeling is focused more
towards achieving high flowrate per path (> 0.3 gpm) and
keeping the coldplate assembly pressure drop is ~9.5 psi range
at 1gpm flowrate per server.

Case I: Serial Flow Option

Fig. 3 illustrates a typical serial flow path option within the
server. Serial flow has fewer connections which is less likely to
have leakage issues. Moreover, a single 4x GPU coldplate
design is likely to cost less compared to two 2x or 4 individual
GPU coldplate design. However, due to the serial approach with
all of the flow going through every coldplate, the overall
coldplate assembly pressure drop is higher than the acceptable
range. The 4x flexible coldplate design is more challenging than
the 3x coldplate that was designed in Summit and Sierra
supercomputers which have been the fastest and the most
powerful supercomputers in the world [8] as all 4 GPU
assemblies need to be touched by 4x flexible coldplate.
Fig. 4. Parallel Flow Path Option
EXPERIMENTAL MEASUREMENT
IBM designed a single flexible coldplate (3x coldplate) to
cool 3 GPUs that has 300 W each and the details of this design
is explained in [7, 8]. The pressure drop at 1 gpm is measured
as 5 psi with 30 cm total 1/4” ID hose (the pressure drop across
the 30 cm, 1/4” ID hose is measured as 0.54 psi at 1 gpm) on
inlet and outlet side of the coldplate. Thus, the net pressure drop
of the coldplate without hoses is 4.46 psi and 3.05 psi at 1 gpm
and 0.79 gpm flowrate, respectively.

Fig. 3. Serial Flow Path Option Fig. 5. Coldplate Mechanical Assembly

Case II: Parallel Flow Path Option The pressure drop within the coldplate for different flowrate
Fig. 4 depicts the parallel flow path option. In parallel flow is illustrated in Fig. 6. The total pressure drop within the
option, each of the two parallel paths consists of a CPU and 2x coldplate with different flowrate is mainly from the overall

Authorized licensed use limited to: University of Glasgow. Downloaded on November 01,2020 at 16:25:39 UTC from IEEE Xplore. Restrictions apply.
mechanical design of the coldplate, inlet and outlet diameter of
3x Flexible GPU
the hose, mechanical structure of the barb and riser, necking Elsewhere
(7 mm Copper Cold Plate
32%
regions in the coldplate and microchannel design. Thus, the Pipe, Hoses, etc..)
35%
motivation of this work is to further improvement of the
coldplate structure for optimal thermal performance of high-end
server design.

2 KKA 3 Quick 2 CPU Cold Plates

Connects 19%
14%

Fig. 8. Percentage of Pressure Drop in Serial Flow Path with

Fig. 6. Pressure Drop vs. Flowrate within the Coldplate
As shown in Fig. 7, the 3x coldplate assembly is connected
to 2 CPU coldplate assemblies with 1/4” ID hose through 7 mm
ID copper pipe to demonstrate the total flow path within the
high-end server. The total pressure drop within the 3x coldplate
assembly including the supply and return hoses are measured as
~14 psi in serial flow path at 1gpm and pressure drop within the
GPU, CPU coldplate and KKA 3 quick connects are
individually measured. From these measurements, GPU
coldplate has 4.46 psi, CPU coldplate has 1.35 psi (total 2.7 psi),
and KKA 3 quick connect pair has 1 psi (total 2 psi) pressure
drop. The pressure drop elsewhere (7 mm copper pipe, hoses,
etc..) is also found as 4.8 psi. The percentage of the individual
component’s pressure drop on overall pressure drop is
summarized in Fig. 8.
3x Coldplate
The pressure drop of the Barb and Riser is of further
importance because they exist on both the GPU and CPU
coldplates. The pressure drop elsewhere (7 mm Copper Pipe,
Hoses, etc.) is also the major contribution on the pressure drop
by 1/3th of the overall pressure drop which is similar as 3x
flexible coldplate. As the server level design has standard hose
size and connections, there is less likely further improvement
on these components; thus, the main focus is towards the
optimal coldplate design to decrease the pressure drop with
enhanced thermal performance for high-speed electronics.
COMPUTATIONAL APPROACH
Ansys Icepak is used for the analysis of the thermal
performance of the coldplate assemblies with TIM2 placed on
NVIDIA Volta GPUs that each has FETs, Inductors, TIM and
HBM dissipating total of 400 W. In this work, 3x and 2x
coldplate assemblies with industry standard 1/4” and 3/8” ID
Barb & Riser mechanical components are designed to
understand the effect of the mechanical design components for
optimum thermal performance of the coldplate assembly.
Flexible 2x coldplate is designed with larger neck region area
to possibly reduce the pressure drop across the coldplate after
the experimental and CFD observations of 3x coldplate. Fig. 9
and Fig. 10 illustrates the CFD model of flexible 3x and 2x
coldplate with Barb & Riser assemblies, respectively. The
design parameters for the analyzed three designs in this work
are also summarized in Table 2.
Speed
Max

Outlet
Barb
Riser Cold plate
Micro-Fin Min

Inlet
TIMs
Fig. 7. Experimental Demonstration of Serial Flow Path with y
3x Coldplate z x
The 3x flexible coldplate is approximately 1/3rd of the Volta GPUs with FETS, Inductors and HBM
overall total pressure drop. Thus, it is important to understand
the effect of Barb and Riser and the necking regions on overall Fig. 9. CFD Model of 3x Coldplate with 1/4 “ ID Barb & Riser
pressure drop within the coldplate structure. Assemblies on GPU Assembly

Authorized licensed use limited to: University of Glasgow. Downloaded on November 01,2020 at 16:25:39 UTC from IEEE Xplore. Restrictions apply.
 (A) Flexible 2x Coldplate
Assembly with Each Neck 16.5 mm 16.5 mm 43 mm
Increased Neck Region Region Length

Microfin 0.3 mm 0.3 mm 0.3 mm

Thickness

Microfin 0.4 mm 0.4 mm 0.4 mm

Spacing

Microfin 4.45 mm 4.45 mm 4.45 mm

Height

Microfin Base 1.1 mm 1.1 mm 1.1 mm

(B) Thickness
Barb Riser
Total Microfin 35.3 x 35.3 35.3 x 35.3 35.3 x 35.3
Outlet Cross-section mm mm mm
Neck Region
RESULTS
Inlet 3x Coldplate Design
a. Design I: Barb & Riser Inlet Diameter = 5 mm
y TIMs (1/4” ID), Flowrate=3 lpm
x
The pressure drop at 3 lpm (~0.79 gpm) is measured
experimentally as 3.05 psi within the 3x coldplate assembly.
z
The CFD model developed for the 3x coldplate assembly is also
Volta GPUs with FETS, Inductors and HBM illustrated in Fig. 11. It is found from CFD analysis that
pressure drop across the inlet and outlet barb assembly is ~3.1
Fig. 10. 2x Flexible Coldplate with 3/8” ID Barb & Riser psi, which there is a very good agreement between numerical
Assemblies with Increased Neck Region: (A) CAD Design (B) and experimental analysis. Fig. 12 illustrates the pressure drop
CFD Modeling on NVIDIA Volta GPU Assembly across the coldplate analyzed from CFD results. It is observed
that ~0.17 psi pressure drop is through the first microchannel
Table 2. Design Parameters for Analyzed Designs
sink, ~0.40 psi pressure drop from the second and ~0.3 psi
Design Design I Design II Design III pressure drop is from the third microchannel. At the neck
Parameters region, ~0.60 psi and ~0.70 psi pressure drop are found at the
first and second region, respectively.
Barb ID 1/4” 3/8” 3/8” Fig. 13 shows the flow characteristics of 1/4” ID inlet Barb
(5 mm) (6.35 mm) (6.35 mm) and Riser. The analyzed pressure drop within the microchannels
and the neck regions from CFD are also illustrated in Fig. 13
Barb length 26 mm 26 mm 26 mm (A). The flow within the inlet Barb is observed as turbulent due
to ReD>~2.2x104 at 3 lpm flowrate. Approximately 0.70 psi and
Riser height, 3 mm x 3 mm x 3 mm x 0.23 psi pressure drop are observed at the inlet 1/4” ID inlet and
width, and 13.3 mm x 13.3 mm x 13.3 mm x outlet Barb & Riser assembly, respectively. When water leaves
length 20 mm 20 mm 20 mm the Barb and enters the Riser, the cross-sectional area increases
and a recirculation cell results in the upper side of the riser,
resulting in a flow constriction beneath it and an associated
pressure drop. With also the effect of the sharp flow turn from
Riser Cross- ~ 38 mm2 ~ 38 mm2 ~ 38 mm2 the Barb assembly to the Riser assembly, pressure reaches to
sectional Area maximum locally and drops through the 1st microchannel.

Neck Region 2.1 mm x 2.1 mm x 5.7 mm x Pressure

3rd Microchannel 2nd Microchannel 1st Microchannel Max
height and 9.2 mm 9.2 mm 8.4 mm (Flow Outlet Section) (Middle Section) (Flow Inlet Section)
width

Neck Region ~19.3 mm2 ~ 19.3 mm2 ~ 45.4 mm2

Cross- 2nd Neck 1st Neck
Region Region
sectional Area Min
(A)

Authorized licensed use limited to: University of Glasgow. Downloaded on November 01,2020 at 16:25:39 UTC from IEEE Xplore. Restrictions apply.
 section on the coldplate which is the neck region need to be
Speed further understood to improve the hydrodynamic
Max characteristics.
y
Barb
Barb Speed
Inlet x
Max

(B)
Min
Cold Plate & GPU Riser
Riser
Assembly

Micro-Fin Min

Inlet
Barb Pressure
Max
Fig. 11. CFD Results of 3x Coldplate: (A) Pressure and (B)
Speed Distribution
Pressure (Psi)

Pressure
y Riser
4.35 x
1st Micro-Fin Min
Micro-
channel

2.90
2nd
Micro-
channel
3rd
Micro-
1.45 channel
Distance (mm)
100 200
Fig. 12. Pressure Drop within 3x Coldplate with 1/4” ID Barb
and Riser Assembly
As there is area change at the flow turn, circulation affects
the flow characteristics at such high ReD number. The gradual
enlargement at the inlet Barb section which is the ratio of the
Barb and Riser diameter, approximately 0.37, results in sudden
area change; thus, such sudden area change leads to pressure
loss. The flow at the outlet section do not encounter similar
effects as in the inlet section which results in lower pressure
drop at the outlet section. For instance, the gradual contractions
at the outlet section has typically lower loss coefficient than
gradual enlargement at the inlet section. This pressure drop
difference at the inlet and outlet section is found as ~0.15 psi
from Macroflow analysis. Also, the effect of gravity due to
water flow within the Barb assembly has a component along
with the flow direction leading to increasing the pressure drop
from the analysis of the conservation of momentum along the
flow direction although due to using short Barb length, this
effect is found not significant. However, this affects oppositely
at the outlet section, which results in lower pressure drop. So,
about 0.47 psi which is almost the twice of the pressure drop at
the outlet Barb & Riser itself is observed due to those affects.
Thus, especially inlet Barb and Riser design as well as critical
Fig. 13. Inlet Flow Section of Barb & Riser Flow
Characteristics: (A) Speed Distribution (B) Pressure Drop
It is also observed that the highest pressure drop occurs
within the 2nd microchannel (middle). Very high speed flow due
to the smaller neck region enters to the 2nd microchannel. Thus,
there is a sudden velocity change at the inlet and the exit section
of the 2nd microchannel due to the fact the 2nd microchannel is
between the two neck regions. However, the flow distribution
at the inlet and the outlet of the 1st microchannel region doesn’t
create such sudden velocity change compared to 2nd
microchannel which results in lowest pressure drop at the 1st
microchannel. The high speed flow also entering to 3rd
microchannel right after passing the 2nd neck region and exiting
the flow through the riser section are relatively observed better
flow characteristics within the riser section than the neck region
due to the geometrical differences between riser and the neck
region. Thus, pressure drop is observed to be lower in 3rd
microchannel than 2st microchannel but higher than 1st
microchannel.
Fig. 14 illustrates the percentage of the pressure drop per
assembly component across the coldplate. Neck regions and the
inlet Barb & Riser assembly are observed as causing the
maximum pressure drop by having more than 20 % of the total
pressure drop across the inlet and outlet section of the coldplate.
The pressure drop at the microchannels are not seen significant
onto the overall pressure drop within the coldplate assembly.
Due to sudden area change at the neck regions (neck cross-
sectional area << fin open area), the flow is not recovered fully
after the flow leaves from the neck region which results in high
frictional loss. Thus, the design at the neck region needs to be
improved in order to reduce the pressure drop at the neck
regions.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 01,2020 at 16:25:39 UTC from IEEE Xplore. Restrictions apply.
 Outlet Barb & Riser 1st Microchannel
5%
7% 2st Microchannel
Inlet Barb & Riser 13%
23%
3rd Microchannel
1st Neck Region
19%
2nd Neck Region
23%
Fig. 14. Percentage of Pressure Drop per Assembly
Component Across the 3x Coldplate with 1/4” ID Barb and
Riser Assembly
b. Design II: Barb & Riser Inlet Diameter = 6.35 mm
(3/8” ID), Flowrate=3 lpm
The impact of increasing the hose ID and inlet barb diameter
so as to reduce the overall pressure drop across the coldplate
assembly was simulated. A 6.35 mm (3/8” ID) inlet diameter
hose is used due to industry standard and packaging limitations
to understand the hydrodynamic characteristics. ReD number is
reduced by ~1.7 times lower than design I with area increase at
the Barb section.
The pressure drop at 3 lpm (~0.79 gpm) is found as 2.09 psi
within the coldplate assembly from CFD modeling. The
pressure drop is observed as ~0.17 psi through the first
microchannel, ~0.43 psi from the 2nd microchannel and ~0.30
psi from the 3rd microchannel. Moreover, ~0.25 psi and ~0.30
psi pressure drop are observed at the 1st and 2nd neck region,
respectively. Also, about 0.43 psi and 0.21 psi pressure drop are
found at the inlet 3/8” ID inlet and outlet Barb & riser assembly,
respectively. The percentage of pressure drop per assembly
component across the coldplate is illustrated in Fig. 15.
Outlet Barb & Riser 1st Microchannel
10% 8%
2st Microchannel
Inlet Barb & Riser
21%
3rd Microchannel
2nd Neck Region
14%
1st Neck Region
12%
Fig. 15. Percentage of Pressure Drop per Assembly
Component Across the 3x Coldplate with 3/8” ID Barb and
Riser Assembly
Overall pressure drop is reduced by ~1 psi within the
coldplate compared to design I due to the reduction on ReD
number and the increase in Barb ID as expected. The pressure
drop at the inlet Barb & Riser assembly are reduced by 0.27 psi
and 0.02 psi, respectively which also corresponds to
approximately 29 % of reduction. Similarly, the pressure drop
is reduced by 0.35 psi and 0.40 psi at the 1st and 2nd neck region,
Authorized licensed use limited to: University of Glasgow. Downloaded on November 01,2020 at 16:25:39 UTC from IEEE Xplore. Restrictions apply.
respectively which is also around 60 % of reduction at the neck
region. Thus, the reduction on the total pressure drop is mainly
resulted from the pressure drop in the inlet Barb & Riser
10% assembly and neck regions which is about 1 psi by only
increasing the Barb diameter by about 1.27 times. The flow
constriction at the gap area between the Barb and Riser structure
is reduced compared to design I which has 1/4 “ Barb ID. With
also reducing the gradual area enlargement by reducing the ratio
of the inlet Barb ID to the riser width compared to design I, the
flow characteristics is observed better in design II which results
in reducing the pressure drop from 0.7 psi to 0.43 psi at the inlet
Barb and Riser assembly. Also, pressure drop within the
microchannel is not changed significantly compared to design I
as expected due to the same microfin structure and spacing with
the same flowrate.
Thus, increasing the neck region area within the coldplate
could result in lower pressure drop at the same flow conditions
with similar or better thermal performance at the same flowrate
observed earlier from Table 1.
2x Coldplate Design
a. Design III: Barb & Riser Inlet Diameter = 6.35
mm (3/8” ID), Flowrate=3 lpm
As a possible reduction of number of neck regions within the
3x coldplate assembly, 2x coldplate could be designed.
However, the dimensions of the coldplate assembly creates a
challenge within the high-end server due to closely spaced
electronic components on board, wiring and packaging issues.
So, after the investigation of mechanical assemblies near the
coldplate and the GPU assembly, the height of the coldplate’s
neck region is observed to be possibly further increased, which
results in approximately 2.35 times higher neck region area
compared to 3x coldplate assembly.
1st Microchannel 2nd Microchannel Pressure
(Flow Inlet Section) (Flow Outlet Section)
Max
21%
Neck Region Min
14%
1st Microchannel 2nd Microchannel
(Flow Inlet Section) (Flow Outlet Section) Speed
Max
Neck Region
Min
Fig 16. Overview 2x Coldplate Assembly: Pressure Drop and
Speed Distribution
 Fig. 16 illustrates the CFD modeling of 2x flexible coldplate
with increased neck region. Around 1.42 psi pressure drop
within the 2x coldplate assembly at 3 lpm (~0.79 gpm) flowrate
is observed. Approximately 0.20 psi pressure drop is found at
the neck region in 2x coldplate assembly. The length of the neck
region that is between the two microchannel inlet is increased
by about 2.6 times compared to each neck region in the 3x
coldplate assembly. So, it is observed that the major loss term
coefficient due to increase in neck region length and reducing
the neck region width by a small amount increases compared to
3x coldplate. Also, the speed at the neck region area is
decreased by increasing the neck region cross-sectional area by
about 2.35 times compared to 3x coldplate assembly. Thus,
overall pressure drop is decreased by approximately 0.1 psi in
increased neck region 2x coldplate compared to 3x coldplate
with 3/8” ID Barb & Riser assembly.
The pressure drop at the inlet and the outlet Barb & Riser
assembly are observed as 0.43 psi and 0.21 psi, respectively
which is also similar as the 3x coldplate assembly with 3/8” ID
Barb & Riser assembly. Fig. 17 depicts the inlet Barb & Riser
flow pressure drop characteristics analyzed by CFD. Similar
phenomena which some circulation and the flow constriction
due to flow turn at the exit of the Barb section are observed at
the inlet section of the 3/8” ID Barb & Riser assembly. This
results in locally achieving pressure between the Barb & Riser
assembly and the pressure drop starts to reduce through the riser
after the flow passes through a higher area in the riser assembly.
Barb Pressure
Inlet
Riser
Micro-Fin
Fig 17. Inlet Barb & Riser Flow Pressure Drop Characteristics
Also, ~0.17 psi and ~0.39 psi pressure drop are observed
through the first and second microchannel sink, respectively. It
is observed that ~0.35 psi pressure drop can be recovered in
total at the neck regions due to reducing the number of neck
regions and increasing the neck region area compared to 3x
coldplate assembly by 2.35. This shows that about 60 %
pressure drop reduction in total within 3x coldplate neck
regions can be achieved by designing 2x coldplate assembly
with 3/8 “ ID Barb and Riser assembly.
The percentage of pressure drop per assembly component
across the 2x coldplate with 3/8” ID Barb and Riser assembly
is also shown in Fig. 18. It is observed similarly as in 3x
coldplate assembly, inlet Barb & Riser and the outlet section
microchannel (2nd microchannel in 2x coldplate assembly)
cause the highest pressure drop components across the 2x
coldplate assembly.
Authorized licensed use limited to: University of Glasgow. Downloaded on November 01,2020 at 16:25:39 UTC from IEEE Xplore. Restrictions apply.
Outlet Barb and Riser
15%
1st Microchannel
12%
2st Microchannel
Inlet Barb and Riser
31%
31%
Neck Region
11%
Fig 18. Percentage of Pressure Drop per Assembly
Component Across the 2x Coldplate with 3/8” ID Barb and
Riser Assembly
Thermal transport is also analyzed on 2 GPU assemblies
each has 400 W on 2x coldplate assembly and the temperature
profile within the coldplate assembly and the water region is
illustrated in Fig. 19. Approximately ~2 °C bulk heating at 3
lpm (~0.79 gpm) flowrate is observed to the last GPU which
matches very well with the analytical calculation.
1st Microchannel 2nd Microchannel
(Flow Inlet Section) (Flow Outlet Section) Temperature
Max
Max
Min
Neck Region
Fig 19. Temperature Distribution in 2x Coldplate Assembly
Min As it is mentioned in Table 1 for case III that 3 parallel path
option which means the flow per path becomes 0.33 gpm at 1
gpm flowrate per server with the same impedance on each
parallel path can be implemented with 2x coldplate design.
Thus, CFD simulation is also performed for 0.33 gpm per
parallel path and maximum temperature on the downstream
GPU is observed about 4 °C lower than the thermal design point
of 400 W GPU assembly (Throttling point of GPU main die 83
°C) which implies that theoretical and CFD results have in very
good agreement. It is also important to mention that GPU
assembly could also throttle based on HBM die if the chip
stacking in the HBM die could have different internal thermal
resistance due to higher memory bandwidth, which is not
considered in this study.
CONCLUSION
In this paper, we investigate the optimal thermal
performance of coldplate design by analyzing different design
options and mainly focusing on serial and different parallel flow
path options for high-end server design. It is observed different
design options don’t affect significantly maximum temperature
on the last GPU which is the overall thermal design point. We
show that modeling the Barb and Riser assembly used in
coldplate design and associated with the pressure drop and flow
characteristics in the coldplate is very important for design.
Using 3/8” ID Barb & Riser in 3x coldplate assembly compared
to 1/4” ID Barb & Riser can lower the pressure drop to ~2.09
psi from ~3.05 psi at 3 lpm (~0.79 gpm). The pressure drop in
2x coldplate design with 3/8” ID Barb and 2.35 times increased
neck region area is observed as 1.4 psi from the CFD modeling
at 3 lpm (~0.79 gpm). The inlet flow within the Barb assemblies
is observed as turbulent. At such high ReD number, frictional
losses are dramatically affected by the relative roughness and
the geometrical structure of the mechanical components &
assembly. Inlet Barb & Riser assembly and neck region design
are found as most critical components on overall pressure drop
in the coldplate assemblies. Local circulation and the flow
constriction at the sudden area changes result in very high
pressure drop which affects the overall hydrodynamic
characteristics of the coldplate.
ACKNOWLEDGEMENT
The authors would like to acknowledge Chris Tuma for
assisting the mechanical development of the coldplate
assembly.
International Technical Conference and Exhibition on
Packaging and Integration of Electronic and Photonic
Microsystems. American Society of Mechanical Engineers
Digital Collection.
[9] Garimella, S. V., and Sobhan, C. B., Transport in
Microchannels—A Critical Review, Annual Review of Heat
Transfer, vol. 13, 2003.
[10] Xuan, Y., & Li, Q. (2003). Investigation on convective
heat transfer and flow features of nanofluids. J. Heat transfer,
125(1), 151-155.
[11] Owhaib, W., & Palm, B. (2004). Experimental
investigation of single-phase convective heat transfer in circular
microchannels. Experimental Thermal and Fluid Science, 28(2-
3), 105-110.
[12] Chiu, H. C., Jang, J. H., Yeh, H. W., & Wu, M. S. (2011).
The heat transfer characteristics of liquid cooling heatsink
containing microchannels. International Journal of Heat and
Mass Transfer, 54(1-3), 34-42.
[13] Zhang, H. Y., Pinjala, D., Wong, T. N., Toh, K. C., &
Joshi, Y. K. (2005). Single-phase liquid cooled microchannel
heat sink for electronic packages. Applied Thermal
Engineering, 25(10), 1472-1487.
[14] Garimella, S. V., & Singhal, V. (2004). Single-phase flow
and heat transport and pumping considerations in microchannel
heat sinks. Heat transfer engineering, 25(1), 15-25.

REFERENCES
[1] Salahuddin, S., Ni, K., & Datta, S. (2018). The era of hyper-
scaling in electronics. Nature Electronics, 1(8), 442-450.
[2] Liao, X. K., Lu, K., Yang, C. Q., Li, J. W., Yuan, Y., Lai,
M. C., ... & Shen, J. (2018). Moving from exascale to zettascale
computing: challenges and techniques. Frontiers of
Information Technology & Electronic Engineering, 19(10),
1236-1244.
[3] Fan, Z., Qiu, F., Kaufman, A., & Yoakum-Stover, S. (2004,
November). GPU cluster for high performance computing. In
Proceedings of the 2004 ACM/IEEE conference on
Supercomputing (p. 47). IEEE Computer Society.
[4] Moldoveanu, A. (2008). Highly-scalable server for massive
multi-player 3D virtual spaces based on multi-processor
graphics cards. Annals of DAAAM & Proceedings, 899-901.
[5] Garimella, S. V., Singhal, V., & Liu, D. (2006). On-chip
thermal management with microchannel heat sinks and
integrated micropumps. Proceedings of the IEEE, 94(8), 1534-
1548.
[6] Schmidt, R. (2004). Challenges in electronic cooling—
opportunities for enhanced thermal management techniques—
microprocessor liquid cooled minichannel heat sink. Heat
Transfer Engineering, 25(3), 3-12.
[7] Tian, S., Takken, T., Schultz, M., Yao, Y., Coteus, P.,
Marroquin, C., O’Connell K, Mahaney HV, Yuksel A, &
Ellsworth, M. (2019). A Single Flexible Coldplate Cools
Multiple Devices. In 2019 18th IEEE Intersociety Conference
on Thermal and Thermomechanical Phenomena in Electronic
Systems (ITherm) (pp. 1313-1320). IEEE.
[8] Yuksel, A., Mahaney, V., Marroquin, C., Tian, S.,
Hoffmeyer, M., Schultz, M., & Takken, T. Thermal and
Mechanical Design of the Fastest Supercomputer of the World
in Cognitive Systems: IBM POWER AC 922. In ASME 2019

Authorized licensed use limited to: University of Glasgow. Downloaded on November 01,2020 at 16:25:39 UTC from IEEE Xplore. Restrictions apply.