CFX-5 Supplementary Theory

6/3/2004 CFX 5
Slide 1 © 2004 ANSYS Canada Ltd.
 Overview
O Transport Equations and Discretization
O Coupled-Multigrid Solver
O Partitioning and Parallelisation

6/3/2004 CFX 5
Slide 2 © 2004 ANSYS Canada Ltd.
 Transport Equations and Discretization

6/3/2004 CFX 5
Slide 3 © 2004 ANSYS Canada Ltd.
 Transport Equations and
Discretization

O Transport Equations
O Finite Volume Method
O Discretization
O Fluxes
O Element Types

6/3/2004 CFX 5
Slide 4 © 2004 ANSYS Canada Ltd.
 Transport Equations

Continuity equation:
∂ρ ∂
+ (ρu j ) = 0
∂t ∂x j
Momentum equation:
⎛ ⎛ ⎞⎞
∂
(ρui ) + ∂ (ρµ j ui ) = − ∂P + ∂ ⎜ µ ⎜ ∂ui + ∂ui ⎟ ⎟ + S
∂t ∂x j ∂xi ∂x j ⎜ eff ⎜ ∂x j ∂x j ⎟ ⎟ ui
⎝ ⎝ ⎠⎠
General transport equation:
∂ ∂ ∂ ⎛⎜ ⎛⎜ ∂φ ⎞⎟ ⎞⎟
(ρφ ) + (ρµ jφ ) = ⎜ Γeff ⎜ ⎟ ⎟ + Sφ
∂t ∂x j ∂x j ⎝ ⎝ ∂x j ⎠ ⎠

6/3/2004 CFX 5
Slide 5 © 2004 ANSYS Canada Ltd.
 Finite Volume Method
r
n Surface A
Gauß theorem:

r r
n
r n
∫ ∇(...)dV = ∫ n (...) dA
Volume V

V A
r
Volume integrated transport equation: n

∂ rr rr
∫
∂t V
ρφ dV + ∫ n Uρφ dA
A
= ∫ n J dA +
A V
∫ Sφ dV
Transient term Convection Diffusion Source term
6/3/2004 CFX 5
Slide 6 © 2004 ANSYS Canada Ltd.
 Discretization
O Element-based finite volume
method
- Generate a mesh
Æ Elements
- Elements are disassembled
into sector volumes which
are grouped around vertices Element
Vertices
Æ Control Volumes
- Transport equations are
solved for control volumes Control
X X volume

X
X

X
X
X X
Integration points (IP)
6/3/2004 CFX 5
Slide 7 © 2004 ANSYS Canada Ltd.
 Control Volume
O Usage of shape factors:
nodes
φ IP = ∑ n φn
F IP

n =1

∂φ nodes

∂x
= ∑ n φn
G IP

n =1
IP
O Assumptions: Vertex Integration point
- Fluxes are constant over the integration point faces of the control
volume
- Values and gradients at the integration points can be calculated
from the vertex values of an element using the shape factors

6/3/2004 CFX 5
Slide 8 © 2004 ANSYS Canada Ltd.
 Convective Fluxes 1

r
U

O Integration over the control volume

rr NIP
r r NIP

∫ nUρφ dA = IP∑=1 n1IP44

A
AIPU IP ρ IP φ IP = ∑ m& IPφ IP
244 3 IP =1
mass flow

O Calculation recipe for ΦIP at the integration point

characterizes the discretization method and order!

6/3/2004 CFX 5
Slide 9 © 2004 ANSYS Canada Ltd.
 Convective Fluxes 2
O Numerical Advection Correction (NAC)

φ IP = φupwind + ∆φ NAC
r
⎛ r ∆φ ⎞ U
= φupwind + β ⎜ ∆x r ⎟
⎝ ∆x ⎠ IP
O Gradient at integration point

∆φ ∆φ ∆φ
r =w ra
+w r
b

∆x IP ∆x Element ∆x Vertex

O Different discretization methods can be selected with the coefficients b, wa

and wb:
- SMUDS, UDS+NAC, UDS+CDS, SOU, QUICK, …

O Variable b ⇒ High Resolution Scheme

6/3/2004 CFX 5
Slide 10 © 2004 ANSYS Canada Ltd.
 Diffusive Fluxes

O Integration over the control volume:

rr NIP
r ⎛ ∂φ ⎞
∫A n J dA = IP∑=1 nIP AIP ⎜⎝ Γ ∂xr ⎟⎠ IP
with

⎛ ∂φ ⎞ ⎛ ∂φ ⎞ nodes
⎜ Γ r⎟ ≈ ΓEL ⎜ r ⎟ ≈ ΓEL ∑ n φ n
IP
G
⎝ ∂x ⎠ IP ⎝ ∂x ⎠ IP n =1

6/3/2004 CFX 5
Slide 11 © 2004 ANSYS Canada Ltd.
 Element Types

O Unstructured meshes (CFX-5)

- Hybrid mesh: mixed element types

O Block-structured grids (CFX-4, CFX-TASCflow)

- Hexahedral elements only

6/3/2004 CFX 5
Slide 12 © 2004 ANSYS Canada Ltd.
 Coupled -Multigrid Solver
Coupled-Multigrid

6/3/2004 CFX 5
Slide 13 © 2004 ANSYS Canada Ltd.
 Overview
O Linear Equation Systems
O Segregated Solution Method
O Coupled Solution Method
O ILU - The Smoother
O Multigrid Theory
O AMG
O List of publications

6/3/2004 CFX 5
Slide 14 © 2004 ANSYS Canada Ltd.
 Linear Equation Systems

Assembly of the linearized transport equations results into

linear equation systems:

aPφ P = a1φ1 + a2φ2 + a3φ3 + ... + a N φ N + bP

ai = f (Convection, Diffusion, ...)

nb nb

∑A
i =1
i , nb φ nb = Bi ∑A
i =1
i , nb ∆ φ nb = Ri

6/3/2004 CFX 5
Slide 15 © 2004 ANSYS Canada Ltd.
 Segregated Solution

O Generic coupled system

(e.g. velocities U and V): ⎛ A uu A uv ⎞ ⎛ U ⎞ ⎛ Bu ⎞
⎜ ⎟⋅⎜ ⎟ = ⎜ ⎟
⎝ A vu A vv ⎠ ⎝ V ⎠ ⎝ Bv ⎠
O Segregated solution:

Outer iterations because of nonlinearity: k →

k+1
Calculate A uu , A uv V k , Bu Solve for Uk+1

Solve for Vk+1 Calculate A vv , A vu U k +1 , Bv

6/3/2004 CFX 5
Slide 16 © 2004 ANSYS Canada Ltd.
 Coupled Solution 1

O Generic coupled system

(e.g. velocities U and V): ⎛ A uu A uv ⎞ ⎛ U ⎞ ⎛ Bu ⎞
⎜ ⎟⋅⎜ ⎟ = ⎜ ⎟
⎝ A vu A vv ⎠ ⎝ V ⎠ ⎝ Bv ⎠
O Coupled solution:

Outer iterations because of nonlinearity: k →

k+1

Calculate A uu , A uv , A vu , A vv , Bu , Bv Solve for Uk+1,Vk+1

6/3/2004 CFX 5
Slide 17 © 2004 ANSYS Canada Ltd.
 Coupled Solution 2
O Coupled solution of linear equation systems:
- The solution variables of all coupled equations are always on the same
time/iteration
- Avoid unphysical over- and undershoots
- Improved robustness:
> Coupling between velocity and pressure
> Coupling of velocity components for rotating systems
> Coupling of the species in a combustion calculation
> Coupling of different phases in a multiphase flow calculation
- More effort per iteration but less iteration required in order to reach
convergence
- 1 ‘big’ matrix instead of several ‘small’ matrices
> Increased memory requirements
- Allows larger time steps

6/3/2004 CFX 5
Slide 18 © 2004 ANSYS Canada Ltd.
 Coupled Solution 3

O Example: Navier-Stokes equations in a single-phase flow

u − momentum : ⎡ Auu Auv Auw Aup ⎤ ⎡ ∆U ⎤ ⎡ RU ⎤

⎢ Avp ⎥⎥ ⎢ ∆V ⎥ ⎢R ⎥
v − momentum : ⎢ Avu Avv Avw ⎢ ⎥ ⎢ V⎥
w − momentum : ⎢ Awu Awv Aww Awp ⎥ ⎢ ∆W ⎥ ⎢ RW ⎥
⎢ ⎥ ⎢ ⎥ ⎢ ⎥
mass : ⎢⎣ Apu Apv Apw App ⎥⎦ ⎣ ∆P ⎦ ⎣ RP ⎦

⎡ Auu 0 0 Aup ⎤
⎢ 0 Avv 0 Avp ⎥⎥
⎢
⎢ 0 0 Aww Awp ⎥
⎢ ⎥
⎢⎣ Apu Apv Apw App ⎥⎦
A Df R
6/3/2004 CFX 5
Slide 19 © 2004 ANSYS Canada Ltd.
 ILU – The Smoother

O Incomplete Lower/Upper-Decomposition
- forward and backward substitution
- ILU factorisation only modifies the main diagonal
Î efficient storage: original matrix + one nodal array
- tends toward a 1-D Tri-Diagonal Matrix Algorithm (TDMA) for
equations with very large coefficients
Î well suited as smoother for AMG solver

A L U
1
1
1
1
1
1
1
1

6/3/2004 CFX 5
Slide 20 © 2004 ANSYS Canada Ltd.
 Multigrid Theory 1
O Constructing a grid hierarchy:
Error to be smoothed
- Fine grids for smoothing short wave
length error components
- Coarse grids for smoothing long
wave length error components
- Coarse grid equations on grid level
l+1 solves for corrections to the fine l
grid solution on grid level l
l+1
O Multigrid advantages:
- Smoothing all components of the l+2
solution error
- Using simple relaxation scheme on l+3
all grids
- Scaling with N log(N) instead of N2

6/3/2004 CFX 5
Slide 21 © 2004 ANSYS Canada Ltd.
 Multigrid Theory 2

Multigrid approaches:
Geometric multigrid (coarsening based on geometry and mesh)
Algebraic multigrid (coarsening based on coefficient matrix)

Multigrid cycles:
different cycle forms available

Fine grid
...
Coarse grid
...
Coarsest grid

V-cycle W-cycle Others ...

6/3/2004 CFX 5
Slide 22 © 2004 ANSYS Canada Ltd.
 AMG - Comparison
O Advantages of ‘simple’ geometric multigrid:
- Simple coarsening rule
- Simple internal data structure to store the matrix
O Disadvantages of ‘simple’ geometric multigrid:
- Performs poorly on problems with anisotropic coefficients
- Requires more complex relaxation schemes
- Hard to apply to unstructured grids
O Advantages of Algebraic Multigrid:
- Coefficient strength is indicator for coarsening
- Simple relaxation scheme (ILU) can be used
- Applicable to unstructured grids
O Disadvantages of Algebraic Multigrid:
- Complex AND solution dependent coarsening rule
- All coarse grids require unstructured matrix structure

6/3/2004 CFX 5
Slide 23 © 2004 ANSYS Canada Ltd.
 AMG - Coarsening

Example: 2D conduction equation with coarsening factor 4

Aspect ratio 1:4

Î coarsening in y-dir.

Aspect ratio 1:1

Î coarsening in x,y-dir.

Aspect ratio 1:1

Î coarsening in x-dir.

y
x

6/3/2004 CFX 5
Slide 24 © 2004 ANSYS Canada Ltd.
 AMG – Coarsening Example

z Convective flow

Mu
ltigr
id-H
i er a
rch
y
6/3/2004 CFX 5
Slide 25 © 2004 ANSYS Canada Ltd.
 AMG – Validation 1
0.00

O Grid refinement:
-1.00
- constant (linear) 25K Nodes
performance with grid 100K Nodes
refinement -2.00
400K Nodes

log 10 (RMS Residual)

-3.00

-4.00

-5.00

-6.00
0 10 20 30 40 50 60 70 80 90 100
Work Unit

6/3/2004 CFX 5
Slide 26 © 2004 ANSYS Canada Ltd.
 AMG – Validation 2

O Sensitivity to grid aspect 0.00

ratio
- Curved duct solved with -1.00
average aspect ratio of 3,
30 and 300

log 10 (RMS Residual)

-2.00
- Comparison of algebraic
multigrid (anisotropic
-3.00
coarsening) with
geometric multigrid
(isotropic coarsening) -4.00

3 GMG
30 GMG
-5.00 300 GMG
3 AMG
30 AMG
300 AMG
-6.00
0 20 40 60 80 100 120
Work Unit

6/3/2004 CFX 5
Slide 27 © 2004 ANSYS Canada Ltd.
 AMG - Publications

O Additive Correction Multigrid (ACM)

- B. .R. Hutchinson and G. D. Raithby,
A Multigrid Method based on the Additional Correction Strategy,
Numerical Heat Transfer, Vol.9, pp. 511-537, 1986

O Algebraic Multigrid Technique (AMG)

- A. Brandt, S. MrCormick, J. Ruge,
Algebraic Multigrid for Sparse Matrix Equations,
Sparsity and Its Applications, Cambridge University Press, 1984
- J. W. Ruge and K. Stüben,
Algebraic Multigrid,
Multigrid Methods, Frontiers in Applied Mathematics, SIAM, 1987

O Coupled Solver
- M. Raw,
A Coupled Algebraic Multigrid Method for the 3D Navier-Stokes
Equations,
Proceedings of the 10th GAMM Seminar, 1995

6/3/2004 CFX 5
Slide 28 © 2004 ANSYS Canada Ltd.
 Partitioning and Parallellization

6/3/2004 CFX 5
Slide 29 © 2004 ANSYS Canada Ltd.
 Overview
O Introduction to parallel computing
O Partitioning
O Architecture
O Validation
O Performance
O Hardware support
O Summary

6/3/2004 CFX 5
Slide 30 © 2004 ANSYS Canada Ltd.
 Parallel Computing 1
O Motivation
- Algorithmic improvements (fast solvers, AMG, …) still not
sufficient to cope with increasing demands for CFD calculations
(huge problems, short turn-around times).

O Solution: Parallel computing!

- Save time by distributing the computational work among different
computers
- Run bigger problems by combining the available memory of
different computers
- Reality: combination of both!
- A “Must” for modern CFD-codes!

6/3/2004 CFX 5
Slide 31 © 2004 ANSYS Canada Ltd.
 Parallel Computing 2
O Main user requirements:
- Speedup and memory scalability
- Support of all features and available on all platforms
- Identical convergence behaviour
- Easy to use and robust (“One button feature”)

O Parallelization concept:
- Domain decomposition
- Support of shared and distributed memory
- No special pre- and post-processing (switching between serial and
parallel without restrictions)
- Completely embedded in GUI
- Single code concept (serial/parallel, master/slave)
- Parallel support of all features is mandatory during solver development
- Usage of platform independent message passing system (PVM, MPI)

6/3/2004 CFX 5
Slide 32 © 2004 ANSYS Canada Ltd.
 Parallel Computing - Examples

6/3/2004 CFX 5
Slide 33 © 2004 ANSYS Canada Ltd.
 Partitioning Methods

O MeTiS

O Recursive coordinate bisection

O Partitioning in a user specified direction

6/3/2004 CFX 5
Slide 34 © 2004 ANSYS Canada Ltd.
 Parallel Flow Chart
user
USER interface MASTER
SLAVE SLAVE
Read and
distribute initial data

SC Establish communicaton

RC
Equation assembly

Time step loop

IPC
Linear solver

WORK
Solution diagnostics

Solution update

Receive and
write final results

6/3/2004 CFX 5
Slide 35 © 2004 ANSYS Canada Ltd.
 Message Passing

Process 1
Data
Application
Level
pack send

Communication ‘Network’
Buffer Buffer
Level
receive
unpack

Application
Level
Process time
Process 2
computing

sending buffer

6/3/2004 CFX 5
Slide 36 © 2004 ANSYS Canada Ltd.
 Validation – Convergence

6/3/2004 CFX 5
Slide 37 © 2004 ANSYS Canada Ltd.
 Indy Car
Surface grid

Pressure distribution
16 Partitions and velocity vectors
in symmetry plane
 Validation - Memory
Scalability
O Test problem with 1.300.000 nodes (hex-grid):

REAL Workspace INTEGER Workspace

350 70
300 60
250 50
106 words

106 words
200 40
150 30
100 20
50 10
0 0
1 Part. 2 Part. 4 Part. 8 Part. 16 Part. 1 Part. 2 Part. 4 Part. 8 Part. 16 Part.

O Most important for workstation clusters!

6/3/2004 CFX 5
Slide 1 © 2004 ANSYS Canada Ltd.
 Validation - Parallel Speedup 1

CFX-5 Parallel Performance

Courtesy Silicon Graphics Inc.
(SGI Origin 2000)

Details of test cases:

Passage: laminar flow, 3.1m hexes

Pelton nozzle: turbulent flow, 6 GGI connections, 28.4m mixed elements, 6.4m nodes
Automotive Pump: two frames of reference, two interfaces, 5.4m mixed elements, 1.3m, nodes
Water Turbine: two frames of reference, two interfaces, 2.8m hexes
F22 plane: transonic external flow, 2.6m tetra elements, 0.43 nodes
Indy car: external flow, 1.9m mixed elements, 0.48 nodes
Two phase: airlift reactor (liquid and gas phases), 2.3m mixed elements, 0.43m nodes

6/3/2004 CFX 5
Slide 2 © 2004 ANSYS Canada Ltd.
 Validation - Parallel Speedup 2

CFX-5 Parallel Performance 64

Courtesy Hewlet Packard Inc. 56 Passage

(HP SuperDome) 48
LeMans

Wall Clock Speedup

Compressible
40

Measured
Ideal
32
HP Machine Ideal
24

16

0
Details of test cases: 0 8 16 24 32 40 48 56 64
Passage: laminar flow, 3.1m hexes Number of CPU's

LeMans car: external flow, 10.3m tet elements, 1.9m nodes

Compressible Passage: Ma=0.6 passage flow, 79.8m hexes, 80.4m nodes

6/3/2004 CFX 5
Slide 3 © 2004 ANSYS Canada Ltd.
 Parallel Hardware Support

O UNIX-Workstations
⇒ Single- or Multi-processor machines
- DEC, HP, IBM, SGI and SUN

O WindowsNT-Workstation
- INTEL-based systems running under WindowsNT 4.0 or
Windows 2000

O Linux-Workstations
- IA32 and IA64 solver binaries

Combined under PVM or MPI

6/3/2004 CFX 5
Slide 4 © 2004 ANSYS Canada Ltd.