Sim2Cool: A Two-Phase Cooling System Simulator and Design Tool

Pritish R. Parida*, Mark. D. Schultz, Timothy Chainer

IBM T. J. Watson Research Center, Yorktown Heights, NY, USA

* prparida@us.ibm.com

,7' - Initial Temperature Difference, [ºC]

ABSTRACT
Q - Empirical coefficient
The development of two-phase cooling for both two- ¨3 - Pressure Drop, [Pa]
dimensional (2D) and three-dimensional (3D) integrated
3 - Pressure, [Pa]
circuits using pumped dielectric coolant has gained recent
4 - Heat Transferred, [W]
attention due to the ability to manage high heat densities,
compatibility with electronics and low cooling energy usage. T - Input/wall heat flux, [W/m2] or [W/cm2]
Development of this approach requires high fidelity and U - Bend Radius, [m]
computationally manageable conjugate thermal models both at 5H - Reynolds number
the device level as well as the system level. Recent studies have TSV - Through Silicon Via
shown micro-scale modeling of in-situ thermal and TTV - Thermal Test Vehicle
hydrodynamic phenomena (e.g. boiling and bubble dynamics) 8$ - Overall Heat Transfer Coefficient, [W/m2-K]
of two-phase flow boiling with good accuracy. This paper [ - Vapor quality, [%]
describes a simulation tool named Sim2Cool which addresses ] - Length along flow direction, [m]
the need for macro-scale system level design simulation for
two-phase cooling. The Sim2Cool simulation tool uses an *UHHN6\PEROV
enthalpy-based / energy balance model for the system Į - Void fraction
components and a reduced order model for device-level model. ȕ - Empirical coefficient
The system simulator tool allows system component Ȗ - Angle of Attack (AoA), [rad] or [degrees]
parameters to be quickly analyzed including; condenser, İ - Void fraction
reservoir and tubing diameter as well as heat dissipation and ߳ - Heat Exchanger Effectiveness
flow rate to quantify the impact on the system coefficient of Ȝ - Thermal conductivity, [W/m-K]
performance (COP). Such tools are critical to accelerating the ȝ - Dynamic viscosity, [Pa-s]
integration of two phase cooling with microelectronics to
ȡ - Density, [kg/m3]
realize performance gains by way of high density electronic
ĭO - Two-phase pressure drop multiplier
packaging as Moore's law on silicon technology advancement
reaches physics limits. 6XEVFULSWV
DEV - Absolute
)U - Frictional
KEY WORDS: Electronics cooling, flow boiling, two-phase
flow modeling, 3D, electrical-thermal models, Co-Design, J - Gravitational
micro pin-fins, radial expanding channels. O - Liquid
OS - Liquid Phase
PRP - Momentum
NOMENCLATURE
S - Pressure
3D - Three Dimensional
VDW - Saturation
a - Empirical coefficient
63 - Single-Phase
C - Empirical coefficient
73 - Two-Phase
COP - Coefficient of Performance
Y - Vapor
&S - Specific heat capacity, [J/kg-K]
' - Diameter, [m]
GK - Hydraulic diameter, [m] INTRODUCTION
(&0 - Embedded two-phase liquid Cooled Over the last couple of decades, computer systems
Microprocessor module performance has grown exponentially. This performance
I - Friction factor improvement, predominantly driven by the scaling of the
* - Mass Flux, [kg/m2-s] transistor to near atomic dimensions, along with an information
J - Gravitational acceleration, 9.81 [m/s2] network supported by data centers throughout the world, has led
H - Height of the liquid column, [m] to the present “Digital Age”. Now, the computer systems in data
h - Enthalpy, [J/kg] centers are facing technology challenges as transistor scaling
reaches an end. To continue scaling computer system
HTC - Heat transfer coefficient, [W/m2-K]
performance, the IT industry is exploring new technologies
IT - Information Technology

978-1-5386-1272-9/$31.00 ©2018 IEEE 271 17th IEEE ITHERM Conference

such as 3D stacking of chips to achieve performance gains
through increased packaging integration density. 3D chip
stacking technology has the potential to enable increased
system performance through close integration and increased
bandwidth of heterogeneous system components such as
processor cores, accelerators and high-density memory [1-3].
However, high power 3D chip stacks require new cooling
solutions to address associated thermal challenges [1-3].
The development of two-phase cooling for both two-
dimensional (2D) and three-dimensional (3D) integrated
circuits using pumped dielectric coolant has gained recent
attention due to the ability to manage high heat densities,
compatibility with electronics and above ambient temperature
operation to achieve very low cooling energy usage [4-9].
Development of this approach requires high fidelity and
computationally manageable conjugate thermal models both at
the device level as well as the system level. Recent studies have
shown micro-scale modeling of in-situ thermal and
hydrodynamic phenomena (e.g. boiling and bubble dynamics)
of two-phase flow boiling with good accuracy [10-14]. This
paper describes a simulation tool named Sim2Cool which
addresses the need for macro-scale system level design
simulation for two-phase cooling. The Sim2Cool simulation
tool uses an enthalpy-based / energy balance model [15, 16] for
the system components including condenser, reservoir and
tubing, and a reduced order model [17, 18] for device-level
model.
In this paper, several use cases of the Sim2Cool system
simulator tool are presented to understand and analyze design
scenarios such as impact of condenser type and size, impact of
using larger or smaller diameter tubing, impact of not having a
reservoir, impact of two dissimilar components in parallel,
impact of heat dissipation and flow rate on COP and other
system operating parameters.
PUMPED TWO-PHASE COOLING SYSTEM
Before describing the simulator system, it is important to
understand the system and associated components for which it
has been developed. Figure 1 shows the cut-away (a) and cross-
sectional schematic (b) of a microprocessor module package
having embedded micro-channels for liquid coolant flow.
Herein referred to as the ECM – Embedded two-phase liquid
Fig. 1. (a) Cut-away view and (b) cross-sectional view of the embedded two-phase liquid cooled 8-core micro-processor module
(ECM). (c) SEM image of cooling channel structures. (d) ECM installed in a commercial server.
Cooled Microprocessor module. Figure 1(c) shows a high-
resolution image of the micro-channels highlighting the central
inlet, inlet flow orifices and radially expanding channels
populated with micro pin-fins array. For the present study, an
8-core IBM POWERTM class microprocessor module was de-
lidded and micro-channels were etched on the backside of the
silicon, followed by a thin glass plate attachment to the silicon
to form the top wall of the micro-channels. Next, a brass
manifold lid was attached to the glass cover plate and to the
organic substrate to contain the fluidic coolant and to provide
fluid supply and return ports. Next, the ECM module was
placed in a commercial server, as shown in Figure 1(d).
The cooling channel structures utilize a significantly
different approach for chip embedded cooling where long
parallel micro-channels are replaced with a central fluid
manifold and radially expanding channels which exit at the
edges of the die. Similar radial arrangements of micro- channels
have been demonstrated in previous studies of cooling plates or
heat exchangers carrying single-phase gas [19, 20] or liquid [21,
22], where this arrangement showed specific pressure drop and
flow stability benefits when operated in a two-phase mode.
Recent numerical studies [23, 24] also reveal that this two-
phase cooling approach could achieve better energy efficiency
and maximum critical heat flux from the resulting reduced flow
path. The etched cooling structures has a radial quadrant
symmetry and includes a central inlet region, 24 inlet orifices
and 24 corresponding radial expanding channels (15-degree
sections each) as shown in Figure 1(c). These etched features
are 120 μm deep and include a pin fin array of 80 μm diameter
pins on 200 μm pitch to simulate through silicon via (TSV)
structures in stacked die arrangement. Additional detail on the
ECM module design and fabrication process can be found in the
referenced paper by Schultz et al. [25].
The coolant (R1234ze) enters the ECM module as sub-
cooled liquid and makes two 90-degree turns to reach the
central inlet region on the chip. Next, the fluid passes through
24 inlet orifices to distribute the flow among the corresponding
24 radial expanding channels (6 per quadrant). A mix of
detailed full-physics and reduced-physics models were used to
model the two-phase flow and heat transfer process to design
and optimize the cooling channels structure including the
central inlet diameter, dimensions of inlet orifices and number
of radial expanding channels. The coolant removes heat from dissipation profiles and flow networks [5, 6, 25], the 1D
the chip as it flows through the radial expanding channels and approach is not applicable. Thus, a reduced physics approach as
partially transitions from liquid phase to vapor phase as it flows described by Parida et al. [17, 18] is utilized for solving the
through the micro-channels. The coolant then exits the ECM as conservation equations of momentum and energy at the
a liquid-vapor mixture. The coolant delivery to and return from evaporator level resulting in a hybrid modeling approach. Note
the ECM is facilitated by the test system shown in Figure 2. The that, the REFPROP software [26] was used for computing the
condenser connected downstream of the ECM, extracts the heat various fluid intrinsic properties.
from the exiting liquid-vapor mixture and condenses the vapor
back to liquid. The liquid coolant then flows into the reservoir
and is pumped back to the ECM module. KEY COMPONENTS OF THE SYSTEM MODEL
The actual pumped two-phase cooling system can be
assembled by bringing together the key components and
connecting them in a specific sequence. For example, in the
system illustrated by Figure 2, the key components are the
pump, test-section, condenser, reservoir and the tubing
connecting each component in a sequence to complete a loop.
The following section describes the approach to model each of
these components.
(i) Equivalent Tubing
The tubing is present throughout the entire cooling loop
connecting one component to the next. Instead of modeling
each tube length and bend radius between two components
individually, an equivalent tube length is considered.
Depending upon the location of the tubing and characteristics
of the associated heat transfer, the pressure drop across the
tubing can be either single phase or two-phase. The approach
proposed by Chen et al. [16] is used to calculate the pressure
drop across the tubing. The total pressure gradient along the
flow direction (]) includes three components – frictional (IU),
gravitational (J) and momentum change due to acceleration or
deceleration (PR), as expressed below:
Fig. 2. Schematic of the test system.
ௗ௉ ௗ௉ ௗ௉ ௗ௉
െቀ ቁ ൌቀ ቁ ൅ቀ ቁ ൅ቀ ቁ (1)
ௗ௭ ௧௢௧௔௟ ௗ௭ ௚ ௗ௭ ௠௢ ௗ௭ ௙௥
SIM2COOL: MODEL DESCRIPTION
Sim2Cool is a simulator tool for modeling the above where,
described pumped two-phase cooling system. This model is a ௗ௉
ቀ ቁ ൌ ሾሺͳ െ ߙሻߩ௟ ൅ ߙߩ௩ ሿ݃ ή ‫ߠ݊݅ݏ‬ (2)
hybrid based upon conservation laws. The basic goal of the two- ௗ௭ ௚
phase cooling system model is to calculate the macroscopic ௗ௉ ௗ ீ మ௫మ ீ మ ሺଵି௫ሻమ
variables, namely the pressure (S), temperature (7) or enthalpy ቀ ቁ ൌ ቂ ൅ ቃ (3)
ௗ௭ ௠௢ ௗ௭ ఘೡ ఈ ఘ೗ ሺଵିఈሻ
(K) and vapor quality ([), at any location in the system [16].
ௗ௉ ௗ௉
There are many approaches to calculate these variables, ቀ ቁ ൌ െ߶௟ଶ ቀ ቁ (4)
ௗ௭ ௙௥ ௗ௭ ௟
however, in order to achieve generality as well as accuracy, the
model is constructed by solving the general one-dimensional where,
(1D) conservation equations of mass, momentum and energy
ௗ௉ ଶ௙೗ ீ మ ሺଵି௫ሻమ
[16]. In this system model, the mass flow rate is assumed to be ቀ ቁ ൌെ (5)
ௗ௭ ௟ ఘ೗ ஽
a known parameter. As a result, the conservation of mass is
automatically satisfied when the same mass flow rate is applied where, the liquid friction factor ݂௟ has the following form:
to every component in the system. Solving momentum ீሺଵି௫ሻ஽ ௡
conservation equation is theoretically equivalent to calculating ݂௟ ൌ ܽ ቀ ቁ (6)
ఓ೗
the pressure drop ΔS, which provides a solution to S; and the
solution of energy conservation equation yields 7RUK and [. in which ܽ and ݊ will take different values for different flow
For any component in the system, such as tubes, pump, regimes. For laminar flows, ܽ ൌ ͳ͸ǡ ݊ ൌ ͳ; for turbulent flows,
condenser, etc., its model can be generalized as a black box, ܽ ൌ ͲǤͲ͹ͻǡ ݊ ൌ ͲǤʹͷ. ߶௟ଶ is the multiplier that has the following
which has inputs ܲ௜௡ , ܶ௜௡ or ݄௜௡ and ‫ݔ‬௜௡ and generates the output form [27]:
ܲ௢௨௧ , ܶ௢௨௧ or ݄௢௨௧ and ‫ݔ‬௢௨௧ according to the characteristics of ஼ ଵ
the component. The present model uses the process described ߶௟ଶ ൌ ͳ ൅ ൅ మ (7)
௑೟೟ ௑೟೟
by Chen et al. [16] to calculate the variables S, 7 and [ across
Where & is determined by the flow regimes of both liquid and
several system components. For some system components such
vapor phases, which is listed in the following table.
as a test-section (evaporator) having complicated heat
Table 1. &value for Lockhart-Martinelli model [27] as the condenser with water as the cold-side coolant and
dielectric fluid R1234ze as the hot-side coolant.
Liquid Vapor C
Turbulent Turbulent 20
Laminar Turbulent 12
Turbulent Laminar 10
Laminar Laminar 5
In Eq. (7), ܺ௧௧ is the Martinelli parameter that is calculated
using the following equation [27]:
ଵି௫ ଴Ǥଽ ఘೡ ଴Ǥହ ఓ೗ ଴Ǥଵ
ܺ௧௧ ൌ ቀ ௫
ቁ ቀ ఘ ቁ ቀఓ ቁ (8)
೗ ೡ
These equations are valid under the following assumptions.
First, the cross-section of the flow has the same area along the
flow direction; second, the velocities of liquid and vapor phases Fig. 4. Schematic illustration of the helically coiled concentric
may be different, but within each phase the velocity is uniform; tube-in-tube heat exchanger (condenser).
third, both two phases are in local thermodynamic equilibrium
[16]. To calculate the overall pressure drop, numerical integral The condenser’s UA was computed using the following:
of equation (1) over the length of the tubing was computed by ଵ ଵ
ܷ‫ ܣ‬ൌ ͳൗቀ ൅ ܴ௜௡௧௙ ൅ ቁ (11)
discretizing the flow length into 100 segments (101 nodes) of ௎஺೏೎ ௎஺ೢ
equal spacing. Figure 3 shows the discretization of the flow where, ܴ௜௡௧௙ is the thermal resistance of the wall between the
channel. Of all the variables effecting the two-phase pressure condenser’s hot-side and cold-side fluids, and ܷ‫ܣ‬ௗ௖ and ܷ‫ܣ‬௪
drop, the vapor quality [ has the strongest influence which are the heat transfer coefficients for the dielectric coolant-side
aggravates further if there is any heat transfer across the tubing and water-side, respectively. The ܷ‫ܣ‬௪ is computed using the
surface. To account for the heat transfer / losses across the correlations developed by Alhajeri et al. [29] and ܷ‫ܣ‬ௗ௖ is
tubing surface and its impact on the total pressure drop, changes computed using the correlations developed by Wongwises and
in enthalpy and thereby changes in vapor quality ([) across each Polsongkram [30]. The initial temperature difference (,7') was
segment along the flow length is calculated. computed as:
Node
No
od N+1=Outlet ‫ ܦܶܫ‬ൌ ܳȀ൫߳݉ሶ௪ ܿ௣ǡ௪ ൯ (12)
Node 1 = Node 2
Inlet Node 3 … Node I Node
I+1
e
… Node N
N
where, Q is the amount of heat exchanged, ߳ is the condenser
Segment 1
Segment 2 … Segment i … Segment N effectiveness derived from the condenser UA and ݉ሶ௪ is the
water-side mass flow rate. Using ,7', the dielectric coolant side
Fig. 3. Discretization of channel along the flow direction [16]. inlet temperature can be computed as:
ܶௗ௖ǡ௜௡ ൌ ܶ௪ǡ௜௡ ൅ ‫ܦܶܫ‬ (13)
For the bends, the following equations are used to compute
the associated frictional pressure drop [28]: The dielectric coolant entering the condenser is assumed to
Δ௉౪౦
be a saturated liquid-vapor mixture and hence, the inlet coolant
ఘ೗ ଶǤଶ
ൌͳ൅ቀ െ ͳቁ ൤൜ͳ ൅ ೝ ൠ ‫ݔ‬ሺͳ െ ‫ݔ‬ሻ ൅ ‫ ݔ‬ଶ ൨ (9) pressure (ܲௗ௖ǡ௜௡ ) is equal to the saturation pressure at ܶௗ௖ǡ௜௡ .
Δ௉ౢ౦ ఘೡ ௄ሺଶା ሻ
ವ
Similar to the tubing, the pressure gradient across the condenser
ீమ
Δܲ୪୮ ൌ ‫݂ܭ‬௟ (10) consists of three components – gravitational, frictional, and
ଶఘ೗
momentum change due to acceleration and deceleration. For the
where, U is the bend radius, ' is the tube diameter, οܲ௧௣ is the gravitational and momentum change components, equations (2)
two-phase pressure drop, οܲ௟௣ is the liquid-phase pressure drop and (3) were used, respectively, while for the frictional
across the 90º bend and . is an empirically derived multiplier. component, correlation proposed by Wongwises and
(ii) Condenser Polsongkram [30] was used. Note that, ܶௗ௖ǡ௜௡ is dependent on
The Condenser is one of the most important components in the UA which is dependent on the average vapor quality in the
the entire pumped two-phase loop. The condenser’s cold side condenser and the average vapor quality is dependent on the
temperature and thermal performance determines the entire pressure drop across the condenser.Depending upon the
loop’s operating conditions as well as influencing the saturation characteristics of the condenser, the fluid coming out of the
temperature at the evaporator. Numerical modeling of the condenser can be either partially condensed or completely
condenser essentially comprises of two inter-related condensed to liquid phase. If the fluid is partially condensed,
components. First, the overall heat transfer coefficient (UA) the pressure drop (ܲௗ௖ǡ௢௨௧ െ ܲௗ௖ǡ௜௡ ) will be negative and the
and second, the pressure drop across the condenser. The overall fluid at the outlet will be a saturated liquid-vapor mixture at the
heat transfer coefficient across the condenser can be either outlet pressure. Thus, the outlet temperature (ܶௗ௖ǡ௢௨௧ ) will be the
obtained from the condenser manufacturer’s data sheets or saturation temperature at the outlet pressure. On the other hand,
computed using its geometric parameters and material if the fluid is completely condensed, the pressure drop can be
properties. In this study, a helically coiled concentric tube-in- either negative or positive depending upon the height of the
tube copper heat exchanger as illustrated in Figure 4, was used condensed liquid phase within the condenser.
 (iii) Reservoir
The reservoir plays an important role in separating the
liquid and the vapor phases, dampening flow oscillations and
stabilizing the system. The fluid separation within the reservoir
introduces discontinuity between its inlet and outlet pressure
while preserving the temperature continuity. In the present
model, it is assumed that the liquid and vapor phases are in
equilibrium within the reservoir. Thus, at the liquid-vapor
interface, the pressure (ܲ௥ ) is equal to the saturation pressure at
the liquid temperature within the reservoir (ܶ௥ ). Assuming no
heat exchange with the ambient room, the liquid temperature
within the reservoir as well as the fluid temperature leaving the
reservoir is equal to the fluid temperature entering the reservoir
from the upstream condenser. That is,
ܶ௥ ሺൌ ܶ௢௨௧ ሻ  ൌ  ܶ௜௡ (14)
The liquid column within the reservoir ensures that the
liquid flowing out of the reservoir is slightly sub-cooled. The
pressure at the outlet of the reservoir can be expressed as:
ܲ௢௨௧ ൌ ܲ௥ ൅ οܲ௟௖ (15)
οܲ௟௖ ൌ ߩ௟ ݃‫ܪ‬ (16)
where, οܲ௟௖ is the pressure head due to the liquid column,
H is the height of the liquid column and is constant for a steady-
state model. Note that the pressure at the reservoir outlet is
independent of the inlet pressure. Once the reservoir outlet
pressure and temperature are known, equations of state can be
used to calculate the key state variables such as enthalpy and
vapor quality at the reservoir outlet.
(iv) Device Under Test / Evaporator
The evaporator (also referred to as the device under test) is
the primary component of the loop where the most complicated
phenomena occurs. The evaporator could be a thermal test
vehicle (TTV) [17] or an ECM module [18] or a cold-plate
attached to a heat dissipating device. In addition to the
calculation of coolant state at evaporator inlet and outlet, the
temperature distribution across the evaporator and/or the heat
dissipating device is also critical. Moreover, each evaporator
design is unique and generalized 1D model cannot be used for
the evaporator modeling. This forces the calculation of the
conservation equations at the evaporator to be done at least two-
dimensionally (2D) or three dimensionally (3D), resulting in an
increased complexity. In the present study, reduced physics
model proposed by Parida et al. [17, 18] is used.
The model of the evaporator consists of two main
components – (i) an electrical model of the heat dissipating
device and (ii) a conjugate thermal model to simulate the fluid
flow and heat transfer behavior. The electrical model captures
the spatially varying, temperature-, operating frequency- and
other such parameters dependent power map. For a passive
device such as a TTV, the power map is only spatially varying
[17], while for an ECM module, the power map is spatially
varying and is temperature-, operating frequency- and supply
voltage-dependent [18].
The conjugate thermal model consists of solid domains and
fluid delivery manifold domains (where the flow is single-
phase) modeled in 3D and fluid flow network where the flow is
in two-phase, modeled in 1D or 2D. In the present study, the
reduced-physics models of the TTV [17] and the ECM [18]
were used. These models have been rigorously validated and
are computationally very efficient compared to detailed physics
model [12]. However, computationally these models are still at
least 1 to 2 orders of magnitude slower than the 1D models of
the other system model components. So, in order to improve the
computation of the evaporator model, higher level abstractions
from the reduced physics model was used. Several operating
scenarios of the TTV and the ECM module were computed
apriori and fed into the system model in the form of correlations
for pressure drop and interpolations for spatial varying device
parameters such as temperature and heat dissipation (in case of
the ECM module).
(v) Inline Pre-heater/Sub-Cooler
In a pumped two-phase cooling loop described in this
paper, the condenser’s cold side temperature determines the
operating pressure of the loop and the condenser effectiveness
determines the operating temperature of the loop. So, to provide
slightly more control on the coolant temperature, an inline pre-
heater or sub-cooler can be used. An inline pre-heater can be
used to increase the temperature of the coolant before the
evaporator to increase the susceptibility of the coolant to boil
within the evaporator. On the other hand, a sub-cooler can be
used to ensure that the coolant is sub-cooled enough not to boil
in the flow manifolds at the evaporator’s inlet section. An inline
pre-heater or an inline sub-cooler is modeled in the same way
as a tubing with heat addition or rejection respectively. By
design, the flow in this component is expected to be single
phase and hence, only single-phase frictional and gravitational
pressure drop is required to be calculated. Additionally, in the
model, a flag is raised if either the inlet or outlet condition of
the inline pre-heater or sub-cooler is in two-phase flow regime.
(vi) Pump
The pump has to provide the pressure head required to
maintain the set mass flow rate while overcoming the pressure
drop associated with the loop. By running several flow rate
cases, the pressure drop – volumetric flow rate (39)
characteristics expected from the pump can be determined and
used for system pump selection. If the pump has already been
selected, the 39 characteristic curve can be used to determine
the operating point and corresponding efficiency and electrical
power consumption of the pump. For the pumped two-phase
system, it is important to ensure that there is no cavitation in the
pump. For that, the net positive suction head (136+) at the
pump inlet is calculated to assess the cavitation potential.
ܰܲܵ‫ ܪ‬ൌ  ൫ܲ௣௨௠௣ǡ௜௡ െ ܲ௦௔௧ ̷ܶ௣௨௠௣ǡ௜௡ ൯ൗ݃ߩௗ௖ǡ௟ (17)
RESULTS AND MODEL VALIDATION
The Sim2Cool system-level simulator was developed in
Matlab® and has a graphical user interface (GUI) through
which a user can interactively change the cooling system
parameters. Figure 5 shows a screen-capture of the simulator
GUI. Using the push-button tabs on the top left side of the
simulator GUI, a user can select a cooling system configuration
out of the four displayed options. Figure 6 shows the currently
available four different cooling system configurations. In
configuration 1 (Figure 6(a)), a pump supplies liquid coolant to
the device under test (DUT) which for the purposes of this
study, is an ECM module. The liquid coolant cools the DUT
and exits out as a liquid/vapor mix which is condensed back to
liquid in the condenser. At the condenser, the heat from the
liquid/vapor mix coolant is transferred to the facility side
coolant loop. The performance of the condenser is determined
by the type of condenser, the facility side coolant and its flow
rate, the refrigerant side coolant, its flow rate and vapor quality,
and the amount of heat exchanged across the condenser. The
performance of the condenser determines the operating
conditions of the entire loop. The condensed liquid from the
condenser flows into the reservoir, where a certain liquid
column is maintained to ensure there is no cavitation occurring
in the pump. In configuration 2 (Figure 6(b)), there is a pre-
heater / sub-cooler between the pump and DUT. In
configuration 3 (Figure 6(c)), there is no pre-heater and no
reservoir. In this case, the liquid column within the condenser
and in the tube between the condenser and the pump is
sufficient to avoid cavitation issue in the pump. In configuration
4 (Figure 6(d)), there are two ECM modules connected in
parallel while rest of the cooling system components are the
same as in configuration 1.
The bottom half of the simulator GUI (Figure 5) shows the
user inputs (purple boxes) and simulator predicted values
(green boxes) grouped by components. The COP values
predicted by the simulator includes the indoor and outdoor loop
pumping powers. The graph on the top right of the GUI plots
the simulator predicted values at different locations in the
cooling system. Using the push-button tabs, a user can select
the type of graph to be displayed. Figure 7 shows all the
available plots for configuration 1 and cooling system
parameters displayed in Figure 5. The solution time for this case
was 2 seconds on a 2.8 GHz / 16 GB RAM system.
Figure 7(a) shows the expected pressure at different
locations in the cooling system. Point A is the location
representing pump outlet, point B is the location representing
DUT inlet, point C represents DUT outlet, point D represents
condenser inlet, point E represents condenser outlet, point F
represents reservoir inlet, point G represents reservoir outlet
and point H represents pump inlet. It can be clearly seen that
the biggest pressure drop occurs across the DUT (that is, B-C)
and the next big pressure drop occurs across the tubing between
the evaporator and the condenser (that is, C-D). The pressure
drop in section A-B is negligible as the fluid flowing in that
section is in liquid phase. The pressure drop across the
condenser (that is, D-E) is also negligible since the two-phase
pressure drop within the condenser is compensated by the
column of condensed liquid within the condenser. Across the
sections E-F and G-H and across the reservoir (F-G), the
coolant is in liquid state and the vertical height difference
between the inlet and outlet results in increased pressure of the
fluid. Since there is no heat transfer in these sections, the
increased fluid pressure results in increased sub-cooling at the
pump inlet (that is, at location H).
The pressure difference (~69 kPa) between the points A
and H is the head provided by the pump at the selected mass
flow-rate of 10 kg/hr and a heat dissipation of ~145 W at the
evaporator. Note that the heat dissipation at the evaporator
(ECM module) is a function of the device temperature,
operating frequency and workload. The average junction
temperature (Tj) reported here is the average temperature of the
ECM cores. From Figure 5, the pumping power required to
operate the pumped two-phase loop is less than 1.7 W at a pump
efficiency of 0.1. The outer-loop requires ~5 W to remove the
heat from the condenser resulting in a COP of over 21.
In Figure 7(a), the pressure recovery due to the vertical
tubing E-F is very similar to the pressure recovery across the
reservoir. Thus, in order to reduce the size and weight of the
system, the reservoir and the tubing G-H can be removed from
the system of configuration 1 resulting in a system of
configuration 3, without significantly effecting the system or
device performance.
Figure 7(b) shows the coolant temperature expected at
different locations in the system. It can be seen that the
temperature at each location is similar except for locations C
and D. The temperature at location D (condenser hot-side inlet)
is roughly 0.9 ºC higher than the cold-side temperature. This
temperature delta is determined by the thermal performance of
the condenser and the amount of heat transferred across the
condenser. For a given amount of heat exchanged across the
condenser, a more effective condenser will result in a lower
temperature delta. On the other hand, a less effective condenser
will result in higher temperature delta. The temperature at
location C (evaporator outlet) is an additional 1 ºC higher. This
temperature delta is directly proportional to the two-phase
pressure drop across the tubing C-D. The higher the two-phase
pressure drop, the higher the temperature delta. Note that within
the operating environment envelope of the dielectric coolant,
the saturation temperature increases monotonically with
coolant pressure. Hence, the saturation (boiling) temperature of
the coolant within the evaporator is equal to or greater than the
temperature at location C. Further, for the current ECM
modules’ flow network most of pressure drop occurs in the inlet
section and across the orifices and the remaining minor portion
of pressure drop occurs in the radial channels. Thus, the
temperature at location C can be used as an indicator of
saturation temperature within the evaporator.
This observation suggests that the conditions or coolant
state at the outlet of the evaporator are critical as they
significantly influence the evaporator thermal performance. A
condenser with poor thermal performance (that is, lower
effectiveness) will result in higher saturation temperature at the
evaporator causing a higher device operating temperature.
Alternately, a long thin tube connecting the evaporator and the
condenser can result in a similar behavior by creating a large
pressure drop between the evaporator outlet and condenser
inlet. Modeling results showed that reducing the diameter of
tubing between locations C and D can increase the pressure
drop in this section by nearly 5X resulting in ~5 ºC increase in
the DUT junction temperature. Hence, proper selection of
components downstream of the evaporator/DUT are very
critical for optimum system and device performance.
Figure 7(c) and Figure 7(d) shows the saturation
temperature and degree of sub-cooling expected at different
locations in the cooling system, respectively. The coolant is 5.8
ºC subcooled at the evaporator inlet, which is sufficient to avoid
any coolant flashing in the inlet section of the evaporator.
Further, it can be observed that the coolant is roughly 1-2 ºC
sub-cooled at the pump inlet which reduces the potential for
cavitation within the pump.
Fig. 5. Graphical User Interface (GUI) of the Sim2Cool System-level Simulator.

Figure 6. Four different cooling system configurations that can be selected in the Sim2Cool System-level Simulator.
Fig. 7. Different graphs that are generated by the Sim2Cool System-level Simulator. (a) Pressure Profile (b) Coolant Temperature
(c) Saturation Temperature (d) Sub-Cooling (e) Vapor Quality (f) System P-h Diagram (g) Zoomed-in P-h diagram (h) Thermal-
map of the DUT.

Figure 7(e) shows the vapor quality variation expected at junction temperature can be expected to be less than 40 ºC for
different locations in the cooling system. A negative vapor a facility side coolant temperature of 25 ºC.
quality means that the coolant is sub-cooled at that location.
Figure 7(f) shows the 3K (pressure-enthalpy) diagram for this Table 2. Key model predictions and comparison against data.
pumped two-phase cooling system. Figure 7(g) shows a Location Data Sim2Cool
zoomed-in image of the 3K diagram indicating the predicted Chip Power [W] 144 144.8
state of the coolant at different locations of the system. The Chip Tj [ᵒC] 37 37.4
pressure is the highest at location A. From location A to B there Inlet Pressure [kPa] 594 592
is slight decrease in pressure at constant enthalpy due to single Pressure Drop [kPa] 63.4 62.6
phase pressure drop without any heat addition/removal. From B DUT Inlet Temperature [ᵒC] 25 25
to C (that is, across the DUT) the pressure drops along with heat DUT Inlet Sub-cooling [ᵒC] 6 5.8
addition into the coolant loop. From C to D, the pressure drops DUT outlet temperature [ᵒC] 27.1 26.9
at constant enthalpy. From D to E (that is, across the condenser) Condenser Inlet Temp [ᵒC] 25.8 25.9
the pressure may drop or recover along with heat removal from Condenser Outlet Temp [ᵒC] 25 25
the coolant. From E to F to G to H, the pressure increases at
constant enthalpy.
A comparison of the model prediction described in Figures
Figure 7(h) shows the thermal map of the ECM module
5 and 7, against the experimental data showed an excellent
expected when operated at 3.4 GHz with a stress-mark
agreement. Table 2 shows the key model predictions and the
workload and cooled by R1234ze flowing at 10 kg/hr. The
corresponding experimental data. For this exemplary case, the
chip power predictions were within 1 Watt (< ±1%) of the
experimental data. For further details on the chip power
prediction accuracy, please refer to Parida el at. [18]. In general,
the chip and cooling system temperature predictions were
observed to be within 0.5 ºC (±4%) of the experimental data.
The pressure at different locations in the system were also
observed to be in excellent agreement (±2%).
CONCLUSIONS
This paper addresses the need for macro-scale system level
design simulation for two-phase cooling. The Sim2Cool
simulation tool described uses an enthalpy-based / energy
balance model for the system components including condenser,
reservoir and tubing, and a reduced order model for device-
level model. The Sim2Cool system-level simulator was
developed in Matlab® and has a graphical user interface (GUI)
through which a user can interactively change the cooling
system parameters.
Using this system simulator tool several what-if scenarios
such as impact of condenser type and size, impact of using
larger or smaller diameter tubing, impact of not having a
reservoir, impact of 2 dissimilar components in parallel, impact
of heat dissipation and flow rate on COP, etc. can be quickly
analyzed. Such tools are critical to accelerating the integration
of two phase cooling with microelectronics to realize
performance gains by way of electronic packaging when
performance gains predicted by Moore's law on silicon
technology advancement are reaching physics limits.
Acknowledgements
This project was supported in part by the U.S. Defense
Advanced Research Projects Agency Microsystems
Technology Office ICECool Applications Program under the
award number FA8650-14-C-7466. Disclaimer: The views,
opinions, and/or findings contained in this article are those of
the author(s) and should not be interpreted as representing the
official views or policies of the Department of Defense or the
U.S. Government. Distribution Statement "A" (Approved for
Public Release, Distribution Unlimited).
The authors would like to thank current and past team
members from IBM Research involved in the ICECool
Applications program for their valuable technical contributions
and discussions. The author would like to thank Ken Plaks
(PM), Avram Bar-Cohen and Kaiser Matin for their technical
support and project guidance.
References:
[1]. T. Mudge, “PicoServer - building a compact energy
efficient multiprocessor”, Int. Conf. on Embedded
Computer Systems: Architectures, Modeling, and
Simulation (SAMOS), July 2008.
[2]. B. Black, M. Annavaram, N. Brekelbaum, J. DeVale, L.
Jiang, G. Loh, D. McCauley, P. Morrow, D. Nelson, D.
Pantuso, P. Reed, J. Rupley, S. Sadasivan, J. Shen, C.
Webb, “Die Stacking (3D) Microarchitecture”, Int. Sym.
on Microarchitecture (MICRO), December 2006.
[3]. J. Jeddeloh, B. Keeth, “Hybrid memory cube – new
DRAM architecture increases density and performance”
Int. Sym. on VLSI Technology (VLSIT), June 2012.
[4]. T. Brunschwiler, B. Michel., H. Rothuizen, U. Kloter, B.
Wunderle, H. Oppermann, and H. Reichl, “Interlayer
Cooling Potential in Vertically Integrated Packages”,
Microsyst. Technol., 15(1):57–74, 2009.
[5]. C. L. Ong, S. Paredes, A. Sridhar, B. Michel, and T.
Brunschwiler. “Radial hierarchical microfluidic
evaporative cooling for 3-d integrated microprocessors.”
4th European Conference on Microfluidics, Limerick,
Ireland, 2014.
[6]. M. Schultz, F. Yang, E. Colgan, R. Polastre, B. Dang, C.
Tsang, M. Gaynes, P. Parida, J. Knickerbocker, T.
Chainer, “Embedded Two-Phase Cooling of Large 3D
Compatible Chips with Radial Channels”, ASME
InterPACK / ICNMM 2015, July 6-9, 2015.
[7]. S. G. Kandlikar, "Review and Projections of Integrated
Cooling Systems for Three-Dimensional Integrated
Circuits," J. Electronic Packaging, 136(2), p. 11, 2014
[8]. S. G. Kandlikar, "History, Advances, and Challenges in
Liquid Flow and Flow Boiling Heat Transfer in
Microchannels: A Critical Review," J. Heat Transf.-
Trans. ASME, 134(3), 2012.
[9]. J. R. Thome, "Boiling in microchannels: a review of
experiment and theory," International Journal of Heat and
Fluid Flow, 25(2), pp. 128-139, 2004.
[10]. P. R. Parida, H. Tsuei, T. J. Chainer, “Eulerian Multiphase
Conjugate Model for Chip-Embedded Micro-Channel
Flow Boiling”, ASME InterPACK / ICNMM, July 6-9,
CA, 2015.
[11]. P. R. Parida and T. J. Chainer, “Eulerian Multiphase
Conjugate Model Development and Validation for Flow
Boiling in Micro-Pin Field”, Proceeding of IEEE ITherm,
May 31-June 3, Las Vegas, NV, 2016.
[12]. P. R. Parida, M. Schultz, O. Ozsun, F. Yang, M. Gaynes,
A. Sridhar, G. McVicker, T. Brunschwiler, T. Chainer,
“Eulerian Multiphase Conjugate Model for Embedded
Two-Phase Liquid Cooled Microprocessor”, IEEE Semi-
Therm Symposium, 2018.
[13]. C. Gorle, P. R. Parida, H. Lee, F. Houshmand, M.
Asheghi, K. Goodson, “Validation Study for VOF
Simulations of Boiling in a Microchannel”, Proceeding of
ASME InterPACK / ICNMM, July 6-9, 2015.
[14]. G. Anjos, “A 3D ALE Finite Element Method for Two-
Phase Flows with Phase Change”, Thesis No. 5426,
EPFL, 2012.
[15]. P. R. Parida, T. Chainer, M. Iyengar, M. David, M.
Schultz, M. Gaynes, V. Kamath, B. Kochuparambil, R.
Simons, R. Schmidt, “System-Level Design for Liquid
Cooled Chiller-less Data Center”, Proceedings of the
ASME 2012 International Mechanical Engineering
Congress & Exposition, November 9-15, Houston, Texas,
USA, 2012.
[16]. L. Chen, F. Yang, P. R. Parida, M. Schultz, T. J. Chainer,
“Enthalpy-based System-Model for Pumped Two-phase
Cooling Systems”, Proceeding of IEEE ITherm, May 31-
June 3, Las Vegas, NV, 2016.
[17]. P. R. Parida, A. Sridhar, M. Schultz, F. Yang, M. Gaynes,
E. Colgan, B. Dang, G. McVicker, T. Brunschwiler, J.
Knickerbocker, T. Chainer, “Modeling Embedded Two-
Phase Liquid Cooled High Power 3D Compatible
 Electronic Devices”, IEEE Semi-Therm Symposium,
2017.
[18]. P. R. Parida, A. Sridhar, A. Vega, M. D. Schultz, M.
Gaynes, O. Ozsun, G. McVicker, T. Brunschwiler, A.
Buyuktosunoglu, T. Chainer, “Thermal Model for
Embedded Two-Phase Liquid Cooled Microprocessor”,
IEEE ITherm Conference, 2017.
[19]. A. Poppe, G. Horvath, G. Bognar, Z. Kohari, M. P. Y.
Desmulliez, M. Rencz, "Experimental study of a radial
micro-channel cooling plate," Proceedings of 7th
Electronics Packaging Technology Conference (IEEE
Cat. No.05EX1233C), pp. 6, 2005.
[20]. M. P. Y. Desmulliez, A. J. Pang, M. Leonard, R. S.
Dhariwal, W. Yu, E. Abraham, G. Bognar, A. Poppe, G.
Horvath, G., Z. Kohari, M. Rencz, D. R. Emerson, R. W.
Barber, O. Slattery, F. Waldron, N. Cordero, "Fabrication
and Characterization of a Low-Cost, Wafer-Scale Radial
Microchannel Cooling Plate," IEEE Transactions on
Components and Packaging Technologies, 32, pp. 20-29,
2009.
[21]. R. Muwanga, I. Hassan, A. Ghorab, "Numerical
investigation of a radial microchannel heat exchanger
with varying cross-sectional channels," Journal of
Thermophysics and Heat Transfer, 22, pp. 321-332, 2008.
[22]. H. Ghaedamini, M. R. Salimpour, A. Campo,
"Constructal design of reverting microchannels for
convective cooling of a circular disc," International
Journal of Thermal Sciences, 50(6), pp. 1051-1061, 2011.
[23]. X. Daguenet-Frick, J. Bonjour, R. Revellin, "Constructal
Microchannel Network for Flow Boiling in a Disc-Shaped
Body," IEEE Transactions on Components and Packaging
Technologies, 33, pp. 115-126, 2010.
[24]. R. Revellin, J. R. Thome, A. Bejan, J. Bonjour,
"Constructal tree-shaped microchannel networks for
maximizing the saturated critical heat flux," International
Journal of Thermal Sciences, 48, pp. 342-352, 2009.
[25]. M. D. Schultz, P. R. Parida, M. Gaynes, O. Ozsun, G.
McVicker, U. Drechsler, T. Chainer, “Microfluidic Two-
Phase Cooling of a High Power Microprocessor – Part A:
Design and Fabrication”, IEEE ITherm Conference, 2017.
[26]. Lemmon, E.W., Huber, M.L., McLinden, M.O, NIST
Standard Reference Database 23: Reference Fluid
Thermodynamic and Transport Properties-REFPROP,
Version 9.1, National Institute of Standards and
Technology, Standard Reference Data Program,
Gaithersburg, 2013.
[27]. Lockhart, R.W., Martinelli, R.C.; Chem. Eng. Prog., Vol.
45. 1949, pp. 39–48.
[28]. Chisholm, D., 1980, “Two-Phase Flow in Bends,” Int. J.
Multiphase Flow, 6, pp. 363–367.
[29]. M. H. Alhajeri, A. M. Koluib, R. Alajmi, S. P. Kalim,
“Effect of Coolant Temperature on the Condensation Heat
Transfer in Air-Conditioning and Refrigeration
Applications”, Experimental Heat Transfer, 22:1, pp. 58-
72, 2009.
[30]. S. Wongwises and M. Polsongkram, “Condensation Heat
Transfer and Pressure Drop of HFC-134a in a Helically
Coiled Concentric Tube-in-Tube Heat Exchanger”, Int. J.
Heat and Mass Transfer, Vol. 49, pp. 4386-4398, 2006.