This action might not be possible to undo. Are you sure you want to continue?
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
Annu. Rev. Fluid. Mech. 1997. 29:123–60
Copyright c 1997 by Annual Reviews Inc. All rights reserved
QUANTIFICATION OF UNCERTAINTY
IN COMPUTATIONAL FLUID
DYNAMICS
P. J. Roache
Ecodynamics Research Associates, Inc., P.O. Box 9229, Albuquerque,
New Mexico 87119
KEY WORDS: numerical uncertainty, veriﬁcation, validation, error estimation, computational
ﬂuid dynamics
ABSTRACT
This reviewcovers Veriﬁcation, Validation, Conﬁrmation and related subjects for
computational ﬂuid dynamics (CFD), including error taxonomies, error estima
tion and banding, convergence rates, surrogate estimators, nonlinear dynamics,
and error estimation for grid adaptation vs Quantiﬁcation of Uncertainty.
Introduction
The focus of this review is the quantiﬁcation of uncertainty, the estimation
or banding of the numerical error of a ﬁnal calculation in computational ﬂuid
dynamics (CFD) and related ﬁelds in computational physics, mathematics, and
engineering. By ﬁnal calculation I mean one considered to be used as is. The
motivation for this calculation is different than that for an error estimate used for
the process of solutionadaptive grid generation. Although the present methods
may be applicable to grid adaptation, and some of the methods described herein
were motivated by that problem, the grid adaptation problemhas vastly different
(though usually unacknowledged) requirements.
The key word is quantiﬁcation of uncertainty, as opposed to vague and all
too common qualitative assessments. Quantiﬁcation of uncertainty may also
involve more than just obtaining a good error estimate; in fact, the more conser
vative procedure based on the gridconvergence index (described below) reports
an error band equal to three times an error estimate. Furthermore, I consider
123
00664189/97/01150123$08.00
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
124 ROACHE
here only a posteriori error estimation, being of the opinion that useful a priori
estimation is not possible for nontrivial ﬂuid mechanics problems.
Policy Statements on Numerical Uncertainty
In 1986, the editors of the ASME Journal of Fluids Engineering (JFE) sawﬁt to
publish a brief policy statement (Roache et al 1986) requiring at least minimal
attention to the quantiﬁcation
1
of numerical accuracy. Although the statement
seemed innocuous and seemed to address an obvious need, it still met signiﬁcant
resistance. (Experience withthe implementationof the policyis giveninRoache
1990.) Since then, other journals have adopted similar explicit policies (ASME
Editorial Board 1994, AIAA 1994, Gresho & Taylor 1994). The JFE has
expanded its original policy, including a prohibition of methods with ﬁrstorder
spatial accuracy (Freitas 1993a, 1995b; see discussion in Shyy & Shindir 1994,
Vanka 1994, Roache 1994b, Freitas 1994; Leonard 1995). An associate editor
of the AIAA Journal (W Oberkampf 1995, personal communication) estimates
that he needs to cite the journal policy in approximately three quarters of his
communications with manuscript authors.
The topic has alsobeenthe subject of several AmericanSocietyof Mechanical
Engineers (ASME) symposia and resulting series of special publications (Celik
& Freitas 1990, Celik et al 1993, Johnson & Hughes 1995), from which the
present article takes its title. Also see the related ASME publications on the
“CFD Triathlons” (Freitas 1993b, 1995a) and other benchmark comparison
exercises, for example, the international HYDROCOINand PSACOINprojects
(OECD 1988, 1990). These exercises give a fair crosssection of present CFD
practices in regard to numerical accuracy and generally are far from edifying.
Oldenburg & Pruess (1995) give an example of qualitatively different ﬂow
revealed by even a cursory grid resolution study that was not discovered by any
participants in the HYDROCOIN code intercomparison exercise.
Veriﬁcation and Validation: Numerical versus Conceptual
Modeling
It is useful at the onset to make the important semantic distinction between
veriﬁcation and validation. Following Boehm (1981) and Blottner (1990),
I adopt the succinct description of veriﬁcation as “solving the equations right,”
and of validation as “solving the right equations.” The code author deﬁnes pre
cisely what partial differential equations (PDEs) are being solved and demon
strates convincingly that they are solved correctly—that is, usually with some
1
Other aspects of conﬁdence building in CFD, including broader issues of code quality
assurance, conﬁrmation, calibration, tuning, and certiﬁcation, are found in Mehta (1989, 1991,
1995), Roache et al (1990), Aeschliman et al (1995), Oberkampf (1994), Oberkampf et al (1995),
Cosner (1995), Melnik et al (1995), and Roache (1995).
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
QUANTIFICATION OF UNCERTAINTY IN CFD 125
order of accuracy and always consistently, so that as some measure of dis
cretization (e.g. the mesh increments) approaches zero, the code produces
a solution to the continuum PDEs; this is veriﬁcation. Whether or not those
equations and that solution bear any relation to a physical problemof interest to
the code user is the subject of validation. In a meaningful sense, a code cannot
be validated; only a calculation (or range of calculations with a code) can be
validated. In my experience, dealing with other than algorithm developers, this
is a difﬁcult concept and requires frequent reiteration.
Another way to make the distinction (i.e. to get to the idea behind the
words, beyond “mere” semantics) is to speak of numerical errors versus con
ceptual modeling errors. An example of conceptual modeling versus numerical
modeling is the assumption of incompressibility. This is clearly a conceptual
modeling assumption. Is it the code builder’s fault, or any criticism of the code
itself, if the user incorrectly applies it? For example, dynamic stall involves
compressibility at a surprisingly low freestream Mach number. Results from
an incompressible code may not agree with experiment very well, but we can
not say that the code fails veriﬁcation because it was applied to a compressible
ﬂow problem, although we may have some sympathy for the user who is fooled
by dynamic stall. But no one would have sympathy for a user who applied
an incompressible ﬂow code to a reentry vehicle at Mach 20. In such a case,
the lack of agreement with experiment is not a code problem but a modeling
problem. The same is true of many practical aspects of applying a CFD code.
The model includes more than the code. “Model” includes conceptual model
ing assumptions (e.g. incompressibility, symmetry, etc). It also includes data
input to the code—e.g. geometry data, which often are not so easy to determine
accurately as one might expect, and boundary conditions and initial conditions.
Incorrect determination of any of these items can lead to failure of validation
of a model, with possibly no criticism of the code.
The question of solution uniqueness always arises with nonlinear equations,
and its position in the veriﬁcationvalidation distinction is important. If, as
stated above, we take veriﬁcation to mean simply that a solution to the con
tinuum PDEs is obtained, then the problem of PDE nonuniqueness is avoided.
This is probably being too easy on the code developer in some cases. Physi
cally inadmissable solutions should be eliminated by the code—for example, a
shocktube simulator shouldbe requiredtoeliminate expansionshocksolutions;
but nature abounds with physically nonunique solutions in ﬂuid dynamics—for
example, hysteresis of airfoil stall and recovery, and bistable ﬂuid ampliﬁers.
If nature cannot decide which solution to produce, we cannot expect more of
a code. Note that obtaining “a solution to the continuum PDEs” may involve
sorting through multiple numerical solutions for nonlinear problems; for exam
ple, see later discussion of Yee et al (1991) and Lafon & Yee (1992). See also
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
126 ROACHE
especially Stephens & Shubin (1981), whose study of Euler solutions indicates
that the multiple numerical solutions converge to the same solution as the grid
is reﬁned.
A related question is the stability of a solution. Although people usu
ally think of artiﬁcial (i.e. numerical artifact) instability as a code difﬁculty,
the equally common situation is artiﬁcial stability, for example, a ﬁrstorder
timedependent code whose artiﬁcial viscosity damps out disturbances so that
accurate calculations of physical ﬂuid instability would require impractically
high resolution. In my opinion, it is asking too much of a code to exactly mimic
stability boundaries except in the limit of →0. For example, it is not a fail
ure of veriﬁcation for a full NavierStokes code to produce steady shearlayer
solutions at Reynolds numbers known to be unstable; an unstable solution is
still a valid solution.
The choice of using veriﬁcation or validation was originally arbitrary and is
now recommended solely because of common developing use. (I have pub
lished articles using the opposite deﬁnition.) In a common English thesaurus,
the terms verify, validate, and conﬁrm are all synonyms, but the words are
used herein, and generally in code qualityassurance (QA), as technical terms
with more contextspeciﬁc meaning. Such technical terms are preferably re
lated to common use, but each term’s technical meaning is deﬁned independent
of common use and in a speciﬁc technical context. This is not a universally
accepted attitude toward semantics. In a widely quoted paper that has been re
cently described as brilliant in an otherwise excellent Scientiﬁc American article
(Horgan 1995), Oreskes et al (1994) think that we can ﬁnd the real meaning of
a technical term by inquiring about its common meaning. They make much of
supposed intrinsic meaning in the words verify and validate and, as in a Greek
morality play, agonize over truth. They come to the remarkable conclusion
that it is impossible to verify or validate a numerical model of a natural sys
tem. Now most of their concern is with groundwater ﬂow codes, and indeed,
in geophysics problems, validation is very difﬁcult. But they extend this to
all physical sciences. They clearly have no intuitive concept of error toler
ance, or of range of applicability, or of common sense. My impression is that
they, like most lay readers, actually think Newton’s law of gravity was proven
wrong by Einstein, rather than that Einstein deﬁned the limits of applicability
of Newton. But Oreskes et al (1994) go much further, quoting with approval
(in their footnote 36) various modern philosophers who question not only
whether we can prove any hypothesis true, but also “whether we can in fact
prove a hypothesis false.” They are talking about physical laws—not just
codes but any physical law. Speciﬁcally, we can neither validate nor inval
idate Newton’s Law of Gravity. (What shall we do? No hazardous waste
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
QUANTIFICATION OF UNCERTAINTY IN CFD 127
disposals, no bridges, no airplanes, no . . . .) See also Konikow & Bredehoeft
(1992) and a rebuttal discussion by Leijnse & Hassanizadeh (1994). Clearly,
we are not interested in such worthless semantics and effete philosophizing,
but in practical deﬁnitions, applied in the context of engineering and science
accuracy.
Inadequate Error Taxonomies
Several taxonomies of errors given in the literature are inadequate and mislead
ing, in my opinion. Not all lists are taxonomies. For example, the list “ﬂora,
fauna, mammals, dogs” is not an adequate biological taxonomy. Mammals are
not separate from fauna but are part of it, as are dogs. But we see this kind of
false taxonomy often in CFD.
For example, socalled gridgeneration errors (Ferziger 1993) are not sepa
rate from discretization errors. For the veriﬁcation of a code or a calculation,
there are no such things as gridgeneration errors [nor are there “errors associ
ated with coordinate transformations” (Ferziger 1993)]. Indeed, bad grids add
to discretization error size, but they do not add new terms. This does not mean
that one grid is as good as another, or that a really bad grid cannot magnify
errors, but only that these socalled gridgeneration errors do not need to be
considered separately from other discretization errors in a gridconvergence
test. If the gridconvergence test is performed, and the errors are shown to
reduce as O(
2
), for example, then all discretization errors are veriﬁed. One
does not need to separately estimate or band the gridgeneration errors.
Likewise for Karniadakis’s (1995) proposed numerical error bar (see also
Vanka 1995), which consists of separately estimated numerical errors from
boundary conditions, computational domain size, temporal errors, and spatial
errors. This is clearly a false taxonomy. Numerical errors at boundaries can
be ordered in [e.g. the various approaches for wall vorticity (Roache 1972)]
or possibly nonordered in [e.g. the boundarylayerlike ∂p/∂n = 0, or
the downstream (outﬂow) boundary location (Roache 1972)]. Ordered errors
will tend to zero as the discretization improves, so that a boundary error from
wall vorticity evaluation need not be considered separately from spatial errors.
That taxonomy (Karniadakis 1995) already includes temporal errors and spatial
errors and computational domain size errors, so that both ordered and non
ordered boundary errors are already counted elsewhere in the taxonomy. Since
the intention in (Karniadakis 1995) is to provide a quantitative breakdown in
the sources of numerical error in an error band, the proposed taxonomy is not
only confused but misleading.
Note, however, that outﬂow boundary errors may prove to be ordered not in
but in 1/l, where l is the distance from the region of interest (e.g. an airfoil)
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
128 ROACHE
to the outﬂow boundary. [See Zingg’s (1993) data, shown in Roache (1994a)
to be ﬁrstorder in 1/l.]
The subject of outﬂow boundary conditions does produce some fuzziness
in categorization of Veriﬁcation vs. Validation, in my opinion. The error can
be ordered, as above, and therefore can be part of veriﬁcation. That is, it is
up to the user (who is doing the conceptual modeling) to estimate or band
the error caused by the position of the outﬂow boundary. But if the code has
some sophisticated outﬂow condition (e.g. a simple vortex condition for Euler
equations), then the distinction is not so clear. Certainly the equations used are
clear, and the code may “solve the equations right” (i.e. veriﬁcation), yet there
exists another benchmark solely from the mathematics (the case with inﬁnite
boundary distance) which could be used to justify the outﬂowcondition without
recourse to physical experiment (which would be associated with validation).
Another example of the same semantic failure arises whenwe consider bench
marking a turbulent boundarylayer code or parabolized NavierStokes (PNS)
code against a Reynoldsstressaveraged full NavierStokes (NS) code. Pre
sume that both codes are convincingly veriﬁed, that is, they correctly solve their
respective equations. Suppose that the PNS code results agree well with the
NS code results for some range of parameters (e.g. including angle of attack).
This agreement is not included in the term veriﬁcation, since the veriﬁcation
of the PNS code has already been completed prior to the NS benchmarking.
Then we could say that the agreement has demonstrated that the PNS code is
“solving the right equations” in one sense, that is, it justiﬁes the use of parabolic
marching equations. Yet to claim validation would be overreaching, since we
have not demonstrated the adequacy of the turbulence model by comparison
with experiment. We have “solved the right equations” only in an intermediate
sense of demonstrating that the PNS equations adequately represent the full
NS equations, but not in the ultimate sense of validation, of “solving the right
physical equations.”
Unfortunately, such “mere semantics” may become of vital interest to code
QA when dealing with regulatory agencies such as the EPA or with legal deﬁ
nitions in a NASA contract.
Taxonomy for Additional Information for Error Estimates
Once we have produced a CFD solution of the governing partial differential
equations, it is clear that we require some additional information in order to
quantitatively estimate the uncertainty or numerical accuracy. The following
taxonomy of sources of this additional information provides a framework for
the discussion. By the word grid, I refer to any measure of discretization
(e.g. Cartesian grid, nonorthogonal grid, number of Fourier modes in a spectral
solution, number of discrete vortices).
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
QUANTIFICATION OF UNCERTAINTY IN CFD 129
Sources of Additional Information for Error Estimation, Given
a CFD Solution of the Governing PDEs on a Grid
A. Additional Solution(s) of the Governing Equations on Other Grids
A.1 Grid Reﬁnement
A.2 Grid Coarsening
A.3 Other Unrelated Grid(s)
B. Additional Solution(s) of the Governing Equations on the Same Grid
B.1 Higherorder Accuracy Solution(s)
B.2 Lowerorder Accuracy Solution(s)
C. Auxiliary PDE Solutions on the Same Grid
D. Auxiliary Algebraic Evaluations on the Same Grid; Surrogate Estimators
D.1 Nonconservation of Conservation Variables
D.2 Nonconservation of Higher Moments
D.3 ZhuZienkiewicztype Estimators
D.4 Convergence of Higherorder Quadratures
The following are brief remarks on this taxonomy, some of which will be
justiﬁed and ampliﬁed in later discussion:
Categories A and B involve the direct, unambiguous evaluation of any error
measure of engineering or scientiﬁc interest. For Category A, no additional
CFD code development or modiﬁcations are required. For Category B, no
additional grid generation is required.
The approach of Category C (e.g. Van Straalen et al 1995) does not simply
involve a local evaluation of something. The key aspect here is that errors are
transported (i.e. advected, diffused). However, a simple local evaluation of
something, without advection, is just what you want to guide solution adap
tation, hence, the different needs of error estimation for solution adaptation
versus quantiﬁcation of uncertainty for a ﬁnal calculation.
Category D, involving algebraic evaluations on the same grid, is cheap, needs
no additional grid generation, and uses no signiﬁcant dynamic memory. How
ever, the error evaluated usually has no direct relation to any error measure of
engineering or scientiﬁc interest.
Grid Reﬁnement and Coarsening
Systematic gridconvergence studies are the most common, most straightfor
ward and arguably constitute the most reliable technique for the quantiﬁcation
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
130 ROACHE
of numerical uncertainty. Unlike the other methods available, this approach
can be used to dependably consider the convergence of any quantity of interest,
as well as the usual L
2
and L
∞
norms.
By gridconvergence studies people usually mean Category A.1 (grid reﬁne
ment), but Category A.2 (grid coarsening) usually would make more sense. If
completely solved solutions are obtained on two grids, presumably the ﬁner
grid solution would be used, so the coarse grid solution could be used to es
timate the error. Whether one reﬁnes or coarsens just depends on which grid
was calculated ﬁrst. So for completely solved solutions, grid reﬁnement and
coarsening are identical.
A disadvantage of Category A methods is that multiplegrid generation is
required. Cartesian grids obviously pose no problem. For boundaryﬁtted
structured grids, the simplest method for grid doubling (halving) is to generate
the ﬁnest grid ﬁrst, using whatever method is preferred (e.g. see Knupp &
Steinberg 1994) and then obtain the coarser grids by removing every other
point (e.g. see Zingg 1993). For noninteger grid reﬁnement (coarsening),
the same generating equations and parameters should be used. See Roache
(1993a, 1994) for further discussion, including nonstructured grid reﬁnement
(see also Pelletier & Ignat 1995) and structured reﬁnement of nonstructured
grids.
Error estimation using other unrelated grids (Category A.3) poses an interest
ing challenge. By unrelated grids I mean two or more grids that are overlapping
but not simply obtained one fromthe other. For example, say grid Ais ﬁner than
grid B in some regions but coarser in others, such as might be obtained in two
steps of an rtype (redistribution) solutionadaptive grid. It would seem that
the two solutions on unrelated grids would provide the additional information
necessary to estimate the uncertainty in either, but to my knowledge a method
for doing so has not been invented.
In order to quantify the uncertainty with systematically reﬁned (coarsened)
grids, we need the convergence rate p to estimate the error. For now, we are
assuming that p is known, that is, that we are using a rigorously veriﬁed code
and are now concerned with quantiﬁcation of the uncertainty of a particular
calculation using two grid solutions. The same methods can be converted to
verify a code, that is, to verify (or determine) p. Code veriﬁcation is discussed
in a later section.
A Method for Uniform Reporting of GridConvergence Studies:
The GridConvergence Index
This sectionpresents a summaryof the mainresults of Roache (1993a,b; 1994a).
The gridconvergence index (GCI) presents a simple method for uniform re
porting of gridconvergence studies without any restriction to integer reﬁnement
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
QUANTIFICATION OF UNCERTAINTY IN CFD 131
(e.g. griddoubling). The GCI is basedongeneralizedRichardsonExtrapolation
involving comparison of discrete solutions at two different grid spacings.
Richardson Extrapolation
Richardson Extrapolation, also known as h
2
extrapolation, the deferred ap
proach to the limit, and iterated extrapolation, was ﬁrst used by Richardson in
1910 and later embellished in 1927. The discrete solutions f are assumed to
have a series representation, in the grid spacing h, of
f = f [exact ] + g
1
h + g
2
h
2
+ g
3
h
3
+· · · (1)
The functions g
1
, g
2
, and so on are deﬁned in the continuum and do not depend
on any discretization. For inﬁnitely differentiable solutions, these functions are
related to all orders to the solution derivatives through the elementary Taylor
series expansions, but this is not a necessary assumption for Richardson Extrap
olation, nor is the inﬁnite series indicated in Equation 1. It is only necessary
that Equation 1 be a valid deﬁnition for the order of the discretization. Thus,
the extrapolation may be valid for ﬁnite element solutions, and so on.
For a secondorder method, g
1
= 0. Then the idea is to combine two
separate discrete solutions, f
1
and f
2
, on two different grids with (uniform)
discrete spacings of h
1
(ﬁne grid) and h
2
(coarse grid), so as to eliminate the
leadingorder error terms in the assumed error expansion, that is, to solve for g
2
at the grid points in Equation 1, substitute this value into Equation 1, and obtain
a more accurate estimate of f [exact ]. The result is the original statement
(Richardson 1927) for h
2
extrapolation:
f = f [exact ] +
_
h
2
2
f
1
−h
2
1
f
2
___
h
2
2
−h
2
1
_
+ H.O.T. (2)
where H.O.T. are higherorder terms. Using the grid reﬁnement ratio r =
h
2
/h
1
, this result can be conveniently expressed in terms of a correction to the
ﬁnegrid solution f
1
, dropping H.O.T.:
f [exact ]
∼
= f
1
+( f
1
− f
2
)/(r
2
−1). (3)
The most common use of this method is with a grid doubling, or halving.
(As noted earlier, these are identical, that is, we just have a coarse grid and a
ﬁne grid.) With r = 2, Equation 3 becomes
f [exact ]
∼
= 4/3 f
1
−1/3 f
2
. (4)
It is often stated that Equation 4 is fourthorder accurate if f
1
and f
2
are
secondorder accurate. Actually, as known by Richardson, this statement is
true only if odd powers are absent in the expansion (Equation 1), which he
achieved by assuming the exclusive use of secondorder centered differences.
If uncentered differences are used (e.g. upstreamweighting of advection terms),
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
132 ROACHE
even if these are secondorder accurate (3point upstream), the h
2
extrapolation
is thirdorder accurate, not fourth. As a practical limitation, even extrapola
tions based on centered differences do not display the anticipated fourthorder
accuracy until the cell Reynolds number Rc is reduced; for the 1D advection
diffusion equation with Dirichlet boundary conditions, Rc < 3 is required
(Roache & Knupp 1993).
Although Richardson Extrapolation is most commonly applied to grid doubl
ing and is often stated to be applicable only to integer grid reﬁnement (e.g.
Conte & DeBoor 1965), this is not required. In order to use Equation 3, it is
necessary to have values of f
1
and f
2
at the same points, which would seem
to require commonality of the discrete solutions and, therefore, integer grid
reﬁnement ratios r (grid doubling, tripling, and so on). However, even in his
1910 paper, Richardson looked forward to deﬁning a continuum f
2
by higher
order interpolation, and in his 1927 paper had a speciﬁc approach worked out.
Ferziger (1993) alludes to this approach with less detail but more generality.
Similarly, Richardson Extrapolation is commonly applied only to obtaining a
higherorder estimate on the coarse grid with h
2
= 2h
1
, but Roache & Knupp
(1993) show how to obtain fourthorder accuracy on all ﬁnegrid points by
simple secondorder interpolation, not of the solution values f
2
, but of the ex
trapolated correction from Equation 4, that is, by secondorder interpolation of
1/3( f
1
− f
2
). The use of simple secondorder interpolation avoids complexities
with nonuniform grids and nearboundary points.
Richardson (1910, 1927) also considered sixthorder extrapolation (using
three grid solutions to eliminate g
2
and g
4
), parabolic and elliptic equations,
staggered grids (then called interpenetrating lattices), rapid oscillations and the
2h wavelength limit, a priori error estimates, singularities, integral equations,
statistical problems, Fourier coefﬁcients, and other noncalculus problems. For
example, Richardson (1927) showed the power of the method in an elegant
example of extrapolating two very crude approximations to a circle, namely
an inscribed square and an inscribed hexagon, to get an estimate of π with
threeﬁgure accuracy.
The usual assumptions of smoothness apply, as well as the assumption
(often veriﬁed) that the local error order is indicative of the global error order.
The extrapolation must be used with considerable caution, since it involves the
additional assumption of monotone truncationerror convergence in the mesh
spacing h. This assumption may not be valid for coarse grids. Also, the ex
trapolation magniﬁes machine roundoff errors and incomplete iteration errors
(Roache 1972). In spite of these caveats, the method is extremely convenient to
use compared to forming and solving direct fourthorder discretizations, which
involve more complicated stencils, wider bandwidth matrices, special con
siderations for nearboundary points and nonDirichlet boundary conditions,
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
QUANTIFICATION OF UNCERTAINTY IN CFD 133
additional stability analyses, and so on, especially in nonorthogonal coordi
nates that generate crossderivative terms and generally complicated equations.
Such an application was given by the present author in Roache (1982).
The method is in fact oblivious to the equations being discretized and to
the dimensionality of the problem and can easily be applied as a postproces
sor (Roache 1982) to solutions on two grids with no reference to the codes,
algorithms, or governing equations that produced the solutions, as long as the
original solutions are indeed secondorder accurate. (By this somewhat abu
sive but common terminology, I mean a solution obtained with a secondorder
accurate method applied in its asymptotic range.) The difference between the
secondorder solution and the extrapolated fourthorder solution is itself a use
ful diagnostic tool, obviously being an error estimator (although it does not
provide a true bound on the error except possibly for certain trivial problems).
It was used very carefully, with an experimental determination rather than an
assumption of the local order of convergence, by de Vahl Davis (1983) in his
classic benchmark study of a model freeconvection problem. See Nguyen &
MaclaineCross (1988) for application to heat exchanger pressure drop coefﬁ
cients. Zingg (1993) applied the Richardson error estimator to airfoil lift and
drag calculations in bodyﬁtted grids. Zingg’s work demonstrates the necessity
of gridconvergence testing even when experimental data are available. In four
of seven cases, experimental agreement was better with coarsegrid calcula
tions than with ﬁne! Also, his data indicate that Richardson Extrapolation can
be applied to the estimation of farﬁeld boundary errors, with the error being
ﬁrst order in the inverse of distance to the boundary (Roache 1994a). Blottner
(1990) has used the same procedure to estimate effects of artiﬁcial dissipation
terms in hypersonic ﬂow calculations.
An important aspect of Richardson Extrapolation is that it applies not only to
pointbypoint solution values but also to solution functionals (e.g. lift coefﬁ
cient C
L
for an aerodynamics problem or integrated discharge for a groundwa
ter ﬂow problem), provided that consistent or higherorder methods are used in
the evaluations (e.g. second or higherorder quadratures for lift) as well as the
basic assumption that the order of the method applies globally as well as locally.
If Richardson Extrapolation is applied to produce (say) fourthorder accurate
grid values, one could in principle calculate a fourthorder accurate functional
like C
L
from the grid values, but it would require careful implementation of
fourthorder accurate quadratures. It is much simpler to apply the extrapo
lation directly to the C
L
s obtained in each grid, requiring only secondorder
quadratures. Indeed, this is a major attraction of Richardson Extrapolation
compared to using fourthorder accurate stencils solved either directly or by
deferred corrections. (Note, however, that the two approaches yield different
answers, although both are fourthorder accurate if done properly.)
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
134 ROACHE
Asigniﬁcant yet overlooked disadvantage of Richardson Extrapolation is that
the extrapolated solution generally is not conservative in the sense of maintain
ing conservation properties (e.g. Roache 1972). This shortcoming could well
dictate that Richardson Extrapolation not be used. For example, if it were used
on the groundwater ﬂow simulations for the Waste Isolation Pilot Plant (WIPP
PA Dept. 1992), it would be more accurate in some norm but would introduce
additional nonconservative (i.e. lack of conservation property) source terms
into the radionuclide transport equation. de Vahl Davis (1983, p. 256) pointed
out the more fundamental problem that the extrapolated solution is “no longer
internally consistent; the values of all the variables do not satisfy a system of
ﬁnite difference approximations.” It is also noteworthy that Richardson (1927)
pointed out that the accuracy of the extrapolation does not apply to arbitrarily
high derivatives of the solution. The extrapolation can introduce noise to the
solution, which, although low level, may decrease the accuracy of the solution
higher derivatives.
Thus, it is not advocated (Roache 1993a,b, 1994a) that Richardson Extrapo
lation necessarily be used to improve the reported solution, since that decision
involves these considerations and possibly others. What is advocated is that,
regardless of whether Richardson Extrapolation is used to improve the solution,
the GCI based on the generalized theory of Richardson Extrapolation should
be used to uniformly report gridconvergence tests.
A ﬁnegrid Richardson error estimator approximates the error in a ﬁnegrid
solution, f
1
, by comparing this solution to that of a coarse grid, f
2
, and is
deﬁned as
E
f i ne
1
=
ε
1 −r
p
, (5)
while a coarsegrid Richardson error estimator approximates the error in a
coarsegrid solution, f
2
, by comparing the solution to that of a ﬁne grid, f
1
,
and is deﬁned as
E
coarse
2
=
r
p
ε
1 −r
p
, (6)
where
ε = f
2
− f
1
, (7)
f
2
= a coarsegrid numerical solution obtained with grid spacing h
2
,
f
1
= a ﬁnegrid numerical solution obtained with grid spacing h
1
,
r = reﬁnement factor between the coarse and ﬁne grid
(r = h
2
/h
1
> 1), and
p = formal order of accuracy of the algorithm.
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
QUANTIFICATION OF UNCERTAINTY IN CFD 135
The actual error A
1
of the ﬁnegrid solution may be expressed (Roache 1993a,
1994a) as
A
1
= E
1
+O
_
h
p+l
, E
2
1
_
, (8)
where l = 1 generally or l = 2 if centered differences have been used. Thus,
E
1
is an ordered estimator and is a good approximation when E
1
1.
As noted above, it is neither necessary nor often desirable to use r = 2, or
grid doubling (halving). Accurate application of these generalized Richardson
based griderror estimators requires that the observed convergence rate equals
the formal convergence rate. This requirement implies that the leadingorder
truncation error term in the error series truly dominates the error.
To account for the uncertainty in these generalized Richardsonbased error
estimates due to various factors and to put all gridconvergence studies on the
same basis as grid doubling with a secondorder method, a safety factor is
incorporated into these estimators and the GCI is deﬁned for ﬁne and coarse
grids as
GCI
f i ne
1
= F
s
E
1
 (9)
GCI
coarse
2
= F
s
E
2
 (10)
F
s
> 1 can be interpreted as a safety factor, since F
s
= 1 gives GCI = E.
That is, the error band reduces to the best estimate of the error, analogous to
a 50% error band of experimental data. I recommended (Roache 1993a, b;
1994a) a more conservative value of F
s
= 3. This value also has the advantage
of relating any gridconvergence study (any r and p) to one with a grid doubling
and a secondorder method (r = 2, p = 2). I emphasize that the GCIs are not
error estimators but are three (or F
s
) times the error estimators, representing
error bands in a loose statistical sense. Ostensibly, if we have a ﬁnegrid and
a coarsegrid solution, we would be expected to use the ﬁnegrid solution, so
reporting of the above ﬁnegrid evaluation of GCI would apply. However, a
practical scenario occurs for which the contrary situation applies, i.e. we use
the coarsegrid solution. Consider a parametric study in which hundreds of
variations are to be run. (For example, consider a 3D timedependent study of
dynamic stall, with perhaps 3 Mach numbers, 6 Reynolds numbers, 6 airfoil
thickness ratios, 3 rotortip designs, and 2 turbulence models, for a total of 648
combinations.) A scrupulous approach would require a gridreﬁnement study
for each case, but most engineers would be satisﬁed with one or a few good
gridreﬁnement tests, expecting, for example, that a grid adequate for a NACA
0012 airfoil could be assumed to be adequate for a NACA0015 airfoil. (In fact,
this is often not justiﬁed by experience. For example, stall characteristics can
be quite sensitive to thickness ratio.) So for the bulk of the stack of calculations,
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
136 ROACHE
we would be using the coarsegrid solution, and we want a GCI for this solution.
That is, we derive a GCI from Equation 5, not as the correction to the ﬁnegrid
solution f
1
, but as the correction to the coarsegrid solution f
2
. In this case,
the error estimate changes and must be less optimistic.
A recent application of the GCI to airfoil calculations is given by Lotz et al
(1995), whichdemonstrates the power of the methodwithout the needfor integer
grid reﬁnement and its application with solutionadaptive grids. Pelletier &
Ignat (1995) have shown that the GCI is applicable, at least in a rough sense,
to unstructured grid reﬁnement.
The motivation for using F
s
> 1 is that F
s
= 1 is analogous to a 50% error
band on experimental data, which is not adequate. My originally recommended
value (Roache 1993a,b; 1994a) of F
s
= 3 is conservative and relates the grid
convergence study to one with a grid doubling with a secondorder method. For
many reasons (see below) this is not unduly conservative when only two grids
are used in the study. However, it is nowclear that F
s
= 3 is overly conservative
for scrupulously performed grid convergence studies using three or more grid
solutions to experimentally determine the observed order of convergence p
(e.g. see the papers in Johnson & Hughes 1995). For such highquality studies,
a modest and more palatable value of F
s
= 1.25 appears to be adequately
conservative. However, for the more common twogrid study (often performed
reluctantly, at the insistence of journal editors) I still recommend the value
F
s
= 3 for the sake of uniform reporting and adequate conservatism.
Code Veriﬁcation
So far, this article has been concerned with error estimation, or veriﬁcation of a
calculation. It has been assumed throughout that the code itself has been veriﬁed
independently. If a code has an error (not an ordered error, but a mistake), then
performance of gridconvergence studies or other errorestimation techniques
will not faithfully produce quantiﬁcation of the uncertainty. However, the
same techniques can be used to verify the code separately, including its order of
convergence, bymonitoringgridconvergence towarda nontrivial exact solution.
It is always possible to obtain a nontrivial exact solution for this procedure, if
necessary, by specifying the exact solution and adding the appropriate forcing
function to the governing PDEs (Steinberg & Roache 1985, Shih et al 1989,
Ethier & Steinmen 1994, Westerink & Roache 1995). The necessary condition
is that the solution be nontrivial, that is, that it have signiﬁcant solution structure
to exercise higherderivative calculations. Signiﬁcantly, it is not at all necessary
that the solution be realistic in any sense. Physically realistic solutions will
be more convincing to the mathematically naive (and therefore have value in
the real world of regulatory agencies, contract management, and so on), but
the code veriﬁcation is just as well accomplished by nonphysical solutions.
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
QUANTIFICATION OF UNCERTAINTY IN CFD 137
(For the alternative philosophy, see Oberkampf 1994, Oberkampf et al 1995,
and Aeschliman et al 1995.) Additional realistic solutions (such as Sudicky &
Frind 1982, corrected) can be used in additional conﬁrmation exercises (Roache
1995).
I claim that this method of code veriﬁcation via systematic gridconvergence
testing (whether or not the GCI is used) is both general and rigorous, and has
proven remarkably sensitive in revealing small and subtle coding errors and
usage inconsistencies, as indicated by the following examples (Westerink &
Roache 1995; Salari et al 1995):
A. In veriﬁcation tests of a commercial groundwater ﬂow code, a ﬁrstorder
error in a single corner cell in a strongly elliptic problemcaused the observed
convergence to be ﬁrstorder accurate (Roache et al 1990).
B. In veriﬁcation tests of our SECO FLOW 2D variable density groundwater
ﬂowcode, ﬁrstorder extrapolation for ghost cell values of only one quantity
(aquifer thickness) along one boundary caused the observed convergence
to be ﬁrstorder accurate (Roache 1993c,d).
C. In contaminant transport calculations (advectiondiffusion + decay, retar
dation, and matrix diffusion), use of a plausible singlegridblock represen
tation for a point source as the grid is reﬁned introduces error in a ﬁnite
volume (or blockcentered ﬁnite difference) formulation. In this cell con
ﬁguration, the cell faces align with the boundaries of the computational
domain, and doubling the number of cells requires the location of the single
cell representing the source to shift by /2. It is to be expected that the
solution accuracy in the neighborhood of the source would be affected. But
surprisingly, the accuracy of timeintegrated discharge across boundaries
far from the source was also degraded to ﬁrstorder accuracy (Salari et al
1995).
D. The observed convergence rate of ostensibly secondorderaccurate turbu
lentboundarylayer codes (Wilcox 1993) can be degraded, apparently by
conditional statements limiting eddy viscosity and deﬁning the boundary
layer edge (DC Wilcox 1995, personal communication,).
E. Airfoil codes can exhibit the expected secondorder convergence rates for
lift and drag but less for moment, possibly because of approximations in
volved in applying quasiperiodicity across cutplanes of a Cgrid
(K Salari 1995, personal communication).
F. When a 2D compressible driven cavity problem is solved with an even
number of vertical grid lines, one must average velocity proﬁles from grid
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
138 ROACHE
lines on each side of the centerline rather than use only one; otherwise, the
convergence will be ﬁrstorder (Roache & Salari 1990).
Note that in Examples C and F apparently ﬁrstorder behavior was obtained
with secondorder veriﬁed codes; the problem in Example C was a subtle con
ceptual modeling error and in Example F was a notsosubtle postprocessing
error. In both cases, the codes had no coding errors and were applied in the
asymptotic range.
Esoteric Errors in Code Veriﬁcation
General CFD or computational physics codes [more general than the simple
Poisson equation in nonorthogonal coordinates of Steinberg & Roache (1985)]
would be difﬁcult to include in a theorem because of esoteric errors. The
difﬁcult aspects of the codes are not algebraic complexity [in Steinberg &
Roache (1985) we convincingly veriﬁed 1800 lines of dense Fortran]; the more
difﬁcult and vexing problems come from option combinations and conditional
differencing. Esoteric errors can arise because of nonlinear ﬂux limiters like
FCT, TVD, hybrid or typedependent differencing, and so on (Steinberg &
Roache 1985).
Code Veriﬁcation versus Veriﬁcation of Calculations
In this review, I distinguish between veriﬁcation of codes and veriﬁcation of
individual calculations. Clearly, code veriﬁcation must be done before the
veriﬁcation of a (real) calculation can proceed. However, this distinction works
both ways; it is not enough to perform a calculation of a new problem with
a veriﬁed code. A code may be rigorously veriﬁed to be (say) secondorder
accurate, but when applied to a new problem, this fact provides no estimate of
accuracy or conﬁdence interval, that is, of the size of the error (as opposed to the
order of the error). Even though the real problem solution may be converging
at the same rate (say secondorder) as the code veriﬁcation problem, it is still
necessary to perform gridconvergence tests in order to band the numerical
error. (It would be preferable to have different words for these two veriﬁcation
activities, but I am at a loss for a clarifying term.) The very important point,
independent of the semantics, is that use of a veriﬁed code is not enough. This
point is probably well recognized by present readers, but it is not universally
so. Especially in the commercial CFD arena, user expectations are often that
the purchase and use of a “really good code” will remove from the user the
obligation of “doing his homework,” that is, the straightforward but tedious
work of veriﬁcation of calculations via systematic gridconvergence studies.
This unrealistic hope is sometimes encouraged by advertising.
Unfortunately, the situation may be even more difﬁcult than this, in that a
new calculation may not even exhibit the veriﬁed order of convergence. If the
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
QUANTIFICATION OF UNCERTAINTY IN CFD 139
code veriﬁcation problem(s) are in some sense close to the real problem of in
terest, the order of convergence (e.g. secondorder) from the code veriﬁcation
exercise may be assumed to hold for the real problem. However, there is a
distinction to be made between the formal order of convergence (as indicated
by analyses of the discretization), the actual asymptotic order of convergence,
and the observed (or apparent) order of convergence. Note that there may be
more than one formal analysis, especially for nonlinear problems, so that formal
order of convergence is not necessarily unique. Then the actual asymptotic rate
of convergence may differ from a formal rate, due to competition of truncation
error terms for a particular problem (Westerink & Roache 1995). This situa
tion typically arises when higher grid resolution reveals higher modes of the
solution, which often occurs in atmospheric and ocean modeling (Westerink
& Roache 1995, Dietrich et al 1990) and could be expected in high Reynolds
number DNS (Direct Numerical Simulation) calculations. It is also possible,
and not altogether unusual, to observe superconvergence, that is, an actual
asymptotic convergence rate higher than a formal rate, due to cancellation (or
partial cancellation) of space and time truncation errors. This observation is
evident in the classical situation of (what are generally) ﬁrst or secondorder
methods for constant velocity advection problems producing the exact answers
for unity Courant number (Roache 1972). Finally, the observed convergence
rate p may differ from the actual asymptotic convergence rate due to failure to
achieve the asymptotic range of grid resolution. This is especially obvious if
solution singularities are present, in which case the method of grid reﬁnement
still can be used to estimate uncertainty, but local values of observed p near the
singularity are required (de Vahl Davis 1983).
The cause of the discrepancy between an observed convergence rate and the
actual asymptotic convergence rate may or may not be revealed by formal anal
ysis. For example, in Roache & Knupp (1993), a method was presented for
completing classical Richardson Extrapolation. The classical method combines
two secondorder solutions on a coarse grid and a doubled ﬁne grid to produce
a fourthorder accurate solution, but only on the coarse grid. The completed
algorithmwas devised to produce a fourthorder solution on the entire ﬁne grid.
However, initial gridconvergence studies indicated only a thirdorder conver
gence rate for the steadystate Burgers equation, even though the formal rate
was fourth order. Further analysis by PM Knupp then showed that, because
of truncation error competition particular to the steady Burgers solution, the
higherorder terms in the error expansion include the Reynolds number param
eter Re, and that the asymptotic range for fourthorder accuracy occurs only for
Re x 3.
For these reasons, it appears to be preferable to verify a code with a non
realistic problem that has nontrivial but regular solution behavior (e.g. a tanh
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
140 ROACHE
function) rather than to use a realistic problem that displays the confusing trun
cation error competition.
Extraction of Convergence Rate from GridConvergence Tests
When using gridconvergence studies to estimate the size of the discretization
error, it is necessary to have at least two grid solutions and an a priori knowledge
of the convergence rate p, for example, to have veriﬁed that the code is second
order accurate ( p = 2). If an exact solution is known or constructed (see
above), it is straightforward to extract the order of convergence p from results
of a systematic gridconvergence test using at least two grid solutions; this
serves to verify a code. However, it is also desirable to verify the order p for an
actual problem, since the observed order of convergence depends on achieving
the asymptotic range, which is problem dependent, and the observed order
may differ from the formal (theoretical) order, or from the order veriﬁed for a
test case, for a variety of reasons (see discussion above and a more complete
discussion in Westerink & Roache 1995). Without an exact solution for the
actual problem, it is necessary to have at least three grid solutions to extract p.
Thus, if there is any suspicion that the grid resolution is not in the asymptotic
range, three grid solutions are necessary to verify (or determine) the observed
rate of convergence and thereby allow estimation of the error.
Blottner (1990) andothers use graphical means, plottingthe error onlogpaper
and extracting the order from the slope. This procedure requires evaluation of
the error itself, which is of course generally not known. If the ﬁnest grid solution
is taken to be the reference value (unfortunately, often called the exact value,
which it obviously is not), then the observed order will be accurate only for
those grids far from the ﬁnest, and the calculated order approaching the ﬁnest
grid will be indeterminate. Blottner (1990) improves on this by estimating the
exact value by Richardson Extrapolation (see also Shirazi & Truman 1989),
but this procedure is somewhat ambiguous since the order is already needed in
order to perform the Richardson Extrapolation.
If the grid reﬁnement is performed with constant r (not necessarily r = 2,
but constant), then the order can be extracted directly from three grid solutions,
without a need for estimating the exact solution, following de Vahl Davis (1983,
p. 254). With subscript 1 indicating the ﬁnest grid in the present notation,
p = ln
_
f
3
− f
2
f
2
− f
1
_
/ ln(r). (11)
Ageneralization of this procedure, not restricted to constant r, is possible using
the generalized theory of Richardson Extrapolation (Roache 1993a, b; 1994a).
For constant r, Equation 19 (typo corrected) of Roache (1994a) may be used
to verify an assumed order p. (It is not necessary to use the GCI itself.) One
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
QUANTIFICATION OF UNCERTAINTY IN CFD 141
calculates
α = GCI
f i ne
12
_
GCI
f i ne
23
. (12)
If α ≈ r
p
, then p is the observed order. However, Equation 12 also requires
r to be constant over the three grid set, and it cannot be used to calculate p
directly since p is implicitly present in the GCIs. The more general procedure
(Roache 1995a) is to solve the equation
ε
23
_
r
p
23
−1
_ = r
p
12
_
ε
12
_
r
p
12
−1
_
_
(13)
for p. This is simple for r constant (not necessarily 2 or integer), giving
p = ln(ε
23
/ε
12
)/ lnr. (14)
(Note that Equation 14 differs slightly from Equation 11 but is equivalent to the
order of the approximation involved.)
If r is not constant during the grid reﬁnement, Equation 13 is transcendental
in p. Usual solution techniques (e.g. NewtonRaphson) can apply, but one
should allow for observed p < 1. This result can happen even for simple
problems at least locally (de Vahl Davis 1983), and in some cases the observed
p < 0 (unfortunately, behavior far away from asymptotic convergence can be
nonmonotone). Also, r ∼ 2 will be easier to solve than r ∼ 1 +δ, and r 2
is probably not of much interest. For wellbehaved synthetic cases, simple
substitution iteration with a relaxation factor ω ∼ 0.5 works well. With ρ =
previous iterate for p, the iteration equation is
p = ωρ +(1 −ω)
ln(β)
ln(r
12
)
(15a)
β =
_
r
ρ
12
−1
_
_
r
ρ
23
−1
_
ε
23
ε
12
. (15b)
Note this form of the iteration gives the exact answer in one step for the case
of r = 2 and ω = 0.
Predicting the Required Grid Resolution
Once p is known with some conﬁdence, one may predict the next level of grid
reﬁnement r
∗
necessary to achieve a target accuracy, expressed as a target error
estimate E
1
or a target GCI
1
called GCI
∗
. With GCI
23
being the value from
Equation 2 for the previous two grids,
r
∗
=
p
_
GCI
∗
/GCI
23
. (16)
This result, of course, depends only on the assumed deﬁnition of order of the
discretization error, that is, only on c = error/
p
and not on the GCI theory
itself.
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
142 ROACHE
Special Considerations for Turbulence Modeling
Special considerations are required for turbulence modeling and for other ﬁelds
with multiple scales. Here, the code theoretical performance can be veriﬁed
(within a tolerance) for a range of parameters but could fail in another range.
It is necessary to get the grid resolution into the asymptotic range in order to
do gridconvergence testing. Virtually any grid is in the asymptotic range for a
simple Laplace equation. For any boundarylayer calculation, it is clear that the
initial (coarse) grid must get some points into the boundary layer. For turbulence
modeling without wall functions (Wilcox 1993, Shirazi & Truman 1989) the
grid must get some points into the wall layer. For turbulence modeling with wall
functions, the grid should not get into the wall layer (Celik & Zhang 1995). In
my interpretation, the wall functions should be viewed as an elaborate nonlinear
boundary condition, and the gridconvergence exercise should be done fromthe
edge of the wall layer out. Similarly, for large eddy simulations (LES), as used
in aerodynamic turbulence research and in atmospheric and ocean modeling
with subgrid turbulence modeling, the grid convergence must not go to zero,
or else the Reynolds stresses will be counted twice, once from the full Navier
Stokes terms and once modeled from the LES terms. Also, the presence of
any switching functions, such as length determinations for the BaldwinLomax
turbulence model (Wilcox 1993) can easily corrupt secondorder convergence
rates.
Finally, the gridresolution requirements are much more demanding for tur
bulent boundary layers, just as laminar boundary layers are much more de
manding than inviscid ﬂows. For example, Claus & Vanka (1992) found that
2.4 million nodes (256 × 96 × 96) did not demonstrate grid independence of
the computed velocity and turbulence ﬁelds of crossﬂow jets.
Special Considerations for Artiﬁcial Dissipation
Special considerations are required for any numerical treatment that effectively
changes the governing PDEs as the grid is reﬁned. Subgrid turbulence mod
eling was mentioned earlier. The other important but troublesome numerical
methods are those that treat shocklike problems with any kind of artiﬁcial
dissipation that depends on the grid size. (Shocklike problems means either
true shocks or simply regions of unresolvable large solution structure involving
solution gradients, curvatures, and various combinations.) This includes both
explicit artiﬁcial dissipation (in which an identiﬁable additional term is added
to the governing PDEs, whether treated explicitly or implicitly in the solution
algorithm) and implicit dissipation introduced locally via nonlinear ﬂux lim
iters (FCT, TVD, ENO, PPM, and so on). If not addressed, these terms corrupt
the convergence behavior. See Kuruvila & Anderson (1985) for discussion of
difﬁculties and pitfalls of doing convergence studies with artiﬁcial dissipation
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
QUANTIFICATION OF UNCERTAINTY IN CFD 143
terms in the equations. The explicit dissipation approach can be treated cor
rectly and rigorously, as in Blottner (1990) and Shirazi & Truman (1989).
The artiﬁcial viscosity of ﬁrstorder upstream differencing (Roache 1972)
requires no special consideration other than its slow convergence. However,
hybrid methods in their various forms cause more problems. By adaptively
changing the weighting given to ﬁrstorder and secondorder stencils, depend
ing on the local cell Reynolds number Rc, these methods by deﬁnition are
ultimately secondorder accurate, but only when the resolution is so ﬁne that
the hybrid algorithm itself is inoperative, typically when Rc < 2. Over the
practical range of resolution of interest, the order is illdeﬁned, but the empiri
cally observed order is, as expected, somewhere vaguely between 1 and 2 (e.g.
see Celik & Zhang 1995). Unfortunately, hybrid methods are commonly used
in commercial codes, wherein the emphasis is often on code robustness at the
sacriﬁce of accuracy. The same illdeﬁned behavior is observed for the class
of AllenSouthwell methods (Roache 1972).
Error Estimation from HigherOrder Solutions
on the Same Grid
Category B.1 methods estimate the accuracy of a base solution by comparison
with “higherorder accuracy solution(s).” (Again, note the somewhat abusive
but common terminology.) Richardson (1908) again scooped modern error
estimationpapers byinventingCategoryB.1, error estimationfromhigherorder
solutions on the same grid, noting that the difference between a secondorder
accurate solution and a fourthorder accurate solution is itself an ordered error
estimator. The higherorder accuracy solution might be obtained via a new
solution of higherorder discretizations using FDM (ﬁnite difference methods),
FVM (ﬁnite volume methods), FEM (ﬁnite element methods), and so on, or by
deferred corrections, compact differences (again direct or deferred corrections,
and so on).
For the technique of error estimation from higherorder solutions on the
same grid, much the same advantages and limitations apply as with the grid
convergence technique. It applies to all point values and functionals (e.g. lift,
drag); x and t errors may be estimated independently or coupled; and the error
estimate includes nonlinear coupling. These methods are not commonly used
because they require additional code capability, unlike gridconvergence tests
(Category A). On the other hand, these Category B methods do not require
additional grid generation.
The development costs of the additional code capability may be minimized
by noting several points. If we were intending to use the higherorder solution
itself, many restrictions and requirements could apply, such as full iterative
convergence, strict conservation, and so on. However, if our only use of the
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
144 ROACHE
higherorder solution is to estimate the error of the base solution, these consid
erations are not so important. The point is that error estimation of, for example,
a secondorder solution using fourth order methods is less demanding numer
ically than obtaining a fourthorder solution for direct use. Also, though not
generally recognized, directional splitting works. (The following development
has not been completed for crossderivative terms, which in any case may re
quire some careful formulation for higherorder stencils and should be veriﬁed
rigorously.)
We use a notation similar to that above for multiple grid solutions: f denotes
the exact solution, f
2
the secondorder accurate solution, and f
4
the fourthorder
accurate solution; C
2x
are the coefﬁcients of the Taylor’s theorem expansion
for the secondorder solution in the xdirection, and so on; and R
2x
are the
remaining terms in the complete secondorder expansion in x, and so on. Then,
in two dimensions,
f = f
2
+C
2x
x
2
+ R
2x
+C
2y
y
2
+ R
2y
(17)
and
f = f
4
+C
4x
x
4
+ R
4x
+C
4y
y
4
+ R
4y
. (18)
Deﬁning the error of the secondorder solution E
2
to be
E
2
= f − f
2
(19)
and substituting for f from Equation 18, we obtain
E
2
= f
4
− f
2
+0(
4
). (20)
This is the basic (Richardson 1908) result that the difference between the
second and fourthorder solutions on the same grid is itself a fourthorder
error estimate for the secondorder solution (a result so obvious that it hardly
deserves to be named a theorem). Likewise obvious, if we deﬁne the error of
the fourthorder solution E
4
to be
E
4
= f − f
4
(21)
and substituting for f from Equation 18, we obtain
E
4
= f
2
− f
4
+0(
2
) (22)
for a secondorder error estimator for the fourthorder solution.
The somewhat less obvious result is for the directional splitting of the higher
order solutions, which can be much easier to implement than a fully directional
higherorder solution. Let f
4x
denote the solution obtained with fourthorder
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
QUANTIFICATION OF UNCERTAINTY IN CFD 145
discretization in x and secondorder in y, and f
4y
denote the solution obtained
with fourthorder discretization in y and secondorder in x. The C
2x
coefﬁcients
are unchanged from the previous deﬁnitions. Then
f = f
4x
+C
4x
x
4
+ R
4x
+C
2y
y
2
+ R
2y
(23)
and
f = f
4y
+C
4y
y
4
+ R
4y
+C
2x
x
2
+ R
2x
. (24)
We nowestimate the difference between f
4x
and f
2
, using Equations 23 and 17,
respectively (if we dropped the higherorder remainder terms at this point, we
would prove the resulting estimator to be secondorder accurate, but by retaining
these terms presently, we show that the estimator is fourthorder accurate)
f
4x
− f
2
= f −C
4x
x
4
− R
4x
−C
2y
y
2
− R
2y
−{ f −C
2x
x
2
− R
2x
−C
2y
y
2
− R
2y
}
= C
2x
x
2
+ R
2x
−C
4x
x
4
− R
4x
.
(25)
Similarly, we estimate the difference between f
4y
and f
2
using Equations 24
and 17, respectively:
f
4y
− f
2
= f −C
4y
y
4
− R
4y
−C
2x
x
2
− R
2x
−{ f −C
2x
x
2
− R
2x
−C
2y
y
2
− R
2y
}
= C
2y
y
2
+ R
2y
−C
4y
y
4
− R
4y
.
(26)
Adding Equations 25 and 26 gives
f
4x
+ f
4y
−2 f
2
= C
2x
x
2
+ R
2x
+C
2y
y
2
+ R
2y
−C
4x
x
4
− R
4x
−C
4y
y
4
− R
4y
.
(27)
In comparison with Equation 17, the ﬁrst four terms on the righthand side of
Equation 27 are identically equal to f − f
2
, without approximation. (That
is, the remainder terms R
2x
and R
2y
are included.) The last four terms on the
righthand side of Equation 27 are O(
4
). With the deﬁnition given in Equation
19, we have the error of the secondorder solution E
2
to be estimated as
E
2
= f
4x
+ f
4y
−2 f
2
+O(
4
). (28)
Thus the estimate for the secondorder solution can be obtained to fourth order
by directionally split fourthorder solutions.
Error Estimation from LowerOrder Solutions
on the Same Grid
Certainly a lowerorder solution on the same grid could be used to estimate the
uncertainty of the higherorder solution (Category B.2), in the same way that
grid coarsening can be used rather than grid reﬁnement, but I know of no such
applications.
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
146 ROACHE
Residual Evaluation and Transport
In one type of Category C method (auxiliary PDE solutions on the same grid)
and in Category D methods as well, residuals of the solution are evaluated.
In the Category C methods, these residuals are used as a source term to a
linearized error equation that mimics the governing equations. That is, the
errors are advected, diffused, and so on. In the various ﬂavors of this approach,
much is made of the distinction between “exact error residual equations” and
“approximate error residual equations.” There is less than meets the eye in this
distinction, because somewhere in the development, additional approximations
are required. Indeed, there can be no exact evaluation of the error of the discrete
pointwise solution by passing it through the continuum operator(s) of the
governing PDEs, since the domain of the PDEs requires more than pointwise
values. The philosophy of FEM, or the underlying conceptual model of FEM
practitioners, is usuallydistinct fromFVMandFDMinthis regard, oftenleading
to disconnects in communications. To most FDM and FVM practitioners, the
numerical solution consists of pointwise, discrete values of the ﬂow variables
(as well as related solution functionals, such as drag coefﬁcients and so on,
obtained by numerical quadratures of the discrete values.) To FDM and FVM
people, the termresidual ordinarily refers to the algebraic residual, the leftovers
from the numerical solutions of the discrete algebraic equations, that will result
from incomplete iterative solution or from accumulated roundoff errors in a
direct matrix solver. Therefore, it makes no sense to FDM or FVM people
to speak about estimating the discretization error of a PDE solution, which
involves realnumber system arithmetic, by evaluating the (algebraic) residual,
which may be made arbitrarily small no matter how crude and inaccurate the
discretization may be. (In fact, the cruder the discretization, the easier it is to
reduce this algebraic residual.)
FEMpractitioners have a different viewpoint. They invariably conceptualize
the numerical solution as more than pointwise, since the selection and con
struction of the interelement basis functions is an essential part of the FEM
methodology. Thus, they conceptualize their numerical solution as a continuum
solution and, thence, are led to consider the (differential) residual resulting from
passing this solution function through the PDE operator. When an FEM author
talks about residuals, he or she refers to what I distinguish here as differential
residuals. (Incidentally, I believe that this difference in attitude and semantics
can be traced to the historical accident that FDM practice grew up with iter
ative solvers, whereas FEM practice grew up with direct solvers.) Note the
major distinction between these approaches: The second (differential) residual
would be nonzero even if the computer used true realnumber arithmetic, but
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
QUANTIFICATION OF UNCERTAINTY IN CFD 147
the ﬁrst (algebraic) residual actually can be driven to machine zero in many
(not necessarily all) algorithms/codes.
Once this useful semantic distinction is made, we can examine the differ
ences in the approach. The practices are indeed different, but the difference is
not as essential as one might be led to think. First of all, we note that most
(moderateorder) FEM solutions, although they produce functions, cannot be
passed through the PDE operator everywhere because they are not smooth at
element boundaries. This limitation can be sidestepped by using the weak or
integral form of the governing equations. Note, however, that conceptualizing
the solution as a continuum function engenders some problems. For example,
the solution may not be truly (locally and globally) conservative, that is, con
serving of mass, momentum, and/or energy, even though the conservation form
(Roache 1972) of the equations was used. Also, when these FEM solutions
are graphed for presentation of results, and often when they are postprocessed
for evaluation of solution functionals such as drag coefﬁcients, the basis func
tions are not used. The quadratures can use the same algorithms used for FDM
solutions or for any pointwise data. Conversely, on the FDM/FVM side, it
is not necessary to conceptualize the solution as consisting of only the dis
crete (node or volume) points. Indeed, a simple methodology for deriving a
ﬁnite difference method involves ﬁtting polynomials through node point values
(e.g. a parabolic ﬁt leads to standard secondorder centered difference sten
cils). Therefore, there is no reason why FEM approaches to error estimation
(either for solution adaptive grids or for quantiﬁcation of uncertainty) cannot
be applied to FDM solutions.
All of this discussion is preamble to the most interesting aspect of residual
evaluation. Regardless of the approach (FEM, FDM, FVM), if a (differential)
residual is tobe evaluatedfor error evaluation, it must be baseduponafunctional
form distinct from that used in the discretization that produced the solution.
If not, the differential residual evaluation will be practically zero, except for
some algebraic garbage caused by incomplete iterative convergence or machine
roundoff. Although arguments may be made about which functional form to
use for the (differential) residual evaluation, they all essentially work for error
estimation and, therefore, the process (a) is somewhat arbitrary, and (b) applies
to FDM and FVM as well as FEM.
Note that each conceptualization of the underlying (subcell) functional form
of the solution leads to a discretization, but the discretization is not necessarily
unique to the conceptualization. For example, the variable coefﬁcient diffusion
problem can be conceptualized as having smooth property variation or piece
wise constant property variation. (In groundwater ﬂowand transport problems,
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
148 ROACHE
the ﬁrst conceptualization typically is held by geophysicists and engineers, and
the second by geologists.) Nevertheless, for a single grid, both conceptualiza
tions can lead to the identical discretization and therefore identical discrete solu
tion. This suggests some tolerance in the construction of subcell solutions. To
put it another way, just because your pointwise solution agrees with experiment
does not necessarily prove the literalness of your underlying conceptualization.
Auxiliary PDE Solutions on the Same Grid
The procedure for error estimation in Category C (auxiliary PDE solutions
on the same grid) then involves selection of an alternate functional form for
the intercell (interelement) solution and evaluation of the (differential) residual.
Again, the derivatives may not exist at element boundaries. Whether one selects
a C
2
smooth function to begin with, or blends (interpolates) the differentials
obtained at element (or cell) centers to produce smooth functions over the
entire region, or puts off such questions to the quadrature rules, etc, is not very
signiﬁcant. The residuals canthenbe usedas local error estimators (CategoryD)
or transported in an error equation (Category C), which is indeed a signiﬁcant
distinction. The major point is that the residual evaluation must involve a
discretization rule different from that used in obtaining the solution. In this
way, these methods are after all not so distinct fromCategory Bmethods, which
use higher (or lower) order stencils to evaluate the error. In Category C, a
higherorder stencil is used to obtain a higherorder solution, but if the stencil
were used only locally to obtain a local residual, it would be like Category D.
The approach in which a rigorously iterated solution is replaced with a less
completely converged solution is intermediate, and the globalness of the error
estimate becomes dependent on the amount and type of iteration.
For examples of the error transport equation approach, see Ferziger (1993)
and Van Straalen et al (1995). In all cases, the error transport equation will be
some linearizedversionof the governingequations. It will be somewhat cheaper
to solve than the original ﬂuid dynamics equations (e.g. full NavierStokes) but,
as expected, will be a less accurate error estimator near separations and so on.
This approach is truly global but not as reliable as Category A or B methods.
Auxiliary Algebraic Evaluations on the Same Grid;
Surrogate Estimators
Category D methods, involving algebraic evaluations on the same grid, are
cheap, need no additional grid generation, and use no signiﬁcant dynamic
memory. However, the error evaluated usually has no direct relation to any
error measure of engineering or scientiﬁc interest. Rather, the error measure
is deﬁned for mathematical convenience (as are the L
2
and L
∞
norms) or by
physical intuition (as is kinetic energy conservation). These measures can be
used as a surrogate for measures of engineering and scientiﬁc interest if and
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
QUANTIFICATION OF UNCERTAINTY IN CFD 149
only if a correlation is established via numerical experimentation over a suite
of problems and range of problem parameters.
Nonconservation of Conservation Variables
Category D.1 (nonconservation of conservation variables) applies only to codes
that do not use fully conservative algorithms for conservation variables. For
example, mass is not identically conserved in most pre1980 boundarylayer
codes, in many FEM codes, as well as other CFD codes. The error estimate
then involves numerical evaluation by quadrature for the erroneous loss or gain
of mass in the computational domain. If the coded algorithm is consistent,
this error will approach zero but only in the limit of → 0 for a nontrivial
problem. (Even for a nominally massconserving full NavierStokes code, the
satisfaction of mass conservation will usually depend on the degree of strict
iterative convergence achieved, but this gives no indication of the discretization
error of the solution, which is our interest here.) Momentum, vorticity, and
internal energy are other possibilities. Note that the evaluation of this error
depends on the accuracy of the quadrature, which probably should be consistent
with the algorithm for solving the PDEs (see also discussion of Category D.4
below), but this does not appear to be a strict requirement. Conservative codes
and algorithms are generally preferred, but nonconservative codes are seen to
offer a readilyevaluatedandunderstood(althoughstill surrogate) error measure.
Nonconservation of Higher Moments
Category D.2 involves evaluation of conservation errors for higherorder mo
ments. In typical turbulent (ReynoldsAveraged) NavierStokes codes, fully
conservative discretization may be used for the primitive variables of mass,
momentum, and internal energy. However, turbulent kinetic energy is typically
not identically conserved. An evaluation of its global and local conservation by
quadrature, including carefully evaluated boundary inﬂowterms and dissipation
(Haworth et al 1993; Chang & Haworth 1995, 1996), then gives a surrogate
indication of general discretization errors. In Chang & Haworth (1995), it
is also used to reliably guide local grid reﬁnement for solutionadaptive grid
generation.
In an earlier section, I noted that, if there is any suspicion that the grid reso
lution is not in the asymptotic range, three grid solutions are necessary to verify
(or determine) the rate of convergence and thereby estimate the error, when the
exact solution is not known. The qualiﬁer used, “when the exact solution is
not known,” may seem redundant when the context is a realistic problem, but
there is an important and useful distinction to be made in regard to Category
D.1 and D.2 methods that are based on conservation errors. Although the con
servation imbalance may not be of direct engineering or scientiﬁc relevance, it
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
150 ROACHE
has the advantage that the exact solution of this quantity is known, namely zero.
Thus, in this surrogate error measure, one requires only a single grid solution
to calculate the error if p is known, and only two grid solutions to extract the
observed convergence rate p (see Chang & Haworth 1996).
ZhuZienkiewiczType Estimators
Category D.3 methods (the ZhuZienkiewicztype estimators) were developed
and intended primarily to guide grid adaptation. See Zhu &Zienkiewicz (1990),
Zienkiewicz & Zhu (1987, 1992), and other FEM methods that have a similar
ﬂavor, such as those of Strouboulis & Oden (1990), Oden et al (1993), Babuska
et al (1994), and Ewing et al (1990). These methods allow the global energy
norm to be well estimated and give good evaluation of local errors (and provide
local estimate of stress accuracy, certainly important for structures problems).
More relevant inthe present context of the quantiﬁcationof uncertainty, these es
timators can be used as surrogate error estimators, and they are relatively cheap.
Although developed for FEM, the ZhuZienkiewicz approach is adaptable
to FDM and FVM (D Pelletier, personal communication), using the concepts
discussed above in the section “Residual Evaluation and Transport.”
The ZhuZienkiewicztype methods share the shortcoming of all surrogate
estimators for ﬂuid dynamics—other than guidance for grid adaptation, there is
little inherent engineering or scientiﬁc interest in the error measure as deﬁned.
Therefore, unless the only interest is mathematics for its own sake, it is neces
sary to establish a correlation of the ZhuZienkiewicz estimator with an error
measure of interest. This can be accomplished only by expensive numerical
experimentation for a given class of problems. For example, a correlation based
on numerical experiments for internal combustion engine modeling would be
unlikely to provide any guidance for external aerodynamics, nor even for a large
range of ﬂow parameters for geometrically similar problems. However, when
one is involvedinextensive suites of calculations, one maybuildupsuchcorrela
tions by experience and arrive at a practical andvery inexpensive error estimator.
This need for establishing a correlation in error measures is also true if one
direct error measure (e.g. obtained by a gridconvergence study) is to be used
as a surrogate for another. For example, in an airfoil calculation, does a 1%
error estimate on C
L
ensure a 1% (or 5%, and so on) error estimate for C
M
?
The correlation providing the (fuzzy) answer will be valid only for a restricted
range of angle of attack, Reynolds number, Mach number, and airfoil class.
Convergence of HigherOrder Quadratures
Category D.4 methods are the simplest, involving convergence of higherorder
quadratures (numerical integration of a known function). For example, quadra
ture for pressure and shear force on the surface leads to evaluation of a drag
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
QUANTIFICATION OF UNCERTAINTY IN CFD 151
coefﬁcient C
D
. For nontrivial problems (i.e. problems with signiﬁcant solution
structure), secondorder quadrature will give a different answer than fourth
order accurate quadrature. As the discretization reﬁnes, these two quadratures
will converge to each other. For coarse grids, the difference may be used as a
surrogate error estimator. The same philosophy can be used pointwise. The
simplest approach ultimately reduces to comparing the calculated pointwise
value of some variable of interest with the value interpolated between the point’s
neighbors. Indeed, these values will converge as the solution converges and,
therefore, can be (and have been) used as surrogate error estimates. However,
the poverty of the concept is shown by the fact that the error estimate is obtained
from a single solution without recourse to the governing PDEs. It is difﬁcult to
take seriously an error estimate for pointwise pressures froman incompressible
NavierStokes code when the same error estimate would result from the same
pointwise values in an M = 20 solution or in Darcy ﬂow in a porous medium.
In fact, since the governing PDEs are not invoked, these error estimators would
be better named simply resolution indicators.
Time Accuracy Estimation
Estimation of the numerical error of the time discretization can be done in
the same manner as the spatial errors, but simpler methods may also be used.
In fact, it is relatively straightforward to estimate the temporal error as the
calculation evolves, and to build a code with a solutionadaptive time step to
control the temporal error to a predetermined level, even when only ﬁrstorder
time differencing is used.
Consider the following inexpensive temporal error estimator (Oden et al
1993; Roache 1993c, d) for a fully implicit (backward) time differencing
method. We write the backward time method for a general equation system in
terms of an operator L (not necessarily linear) as
_
f
n+1
i, j
− f
n
i, j
__
t = L
n+1
. (29)
Generally, the solution will involve an expensive matrix solution for all f
n+1
.
The timeerror estimator uses the difference between backward and forward
time integration, implemented as an extrapolation. The method is very cheap to
implement because it does not require another implicit matrix solution, nor even
another explicit stencil evaluation. It includes the effects of timedependent
boundary conditions and source terms.
Inthe current timestep, Equation29is advancingthe solutionfor f fromtime
level n to time level (n +1) with increment t using fully implicit (backward)
time differencing, so that the operator L is being evaluated at (n + 1). In
the previous timestep, the solution was advanced from (n − 1) relative to the
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
152 ROACHE
current indexing to n with increment t
OLD
and the L evaluated at n. We could
explicitly evaluate L
n
and make a separate, parallel estimate of the values f
n+1
with an explicit step, as in
_
F
n+1
i, j
− f
n
i, j
__
t = L
n
(30)
where F
n+1
signiﬁes the new value of f predicted by the explicit algorithm.
(Since the lefthand side involves values of f only at the location (i, j ) but not
at the neighboring locations (i ± 1, j ) and (i, j ± 1) the equation is explicit,
that is, it does not involve a matrix solution of f at all values of i and j
simultaneously.) The difference between the new value of f
n+1
predicted by
the implicit algorithm and F
n+1
predicted by the explicit algorithm, both of
which predictions are O(t ) accurate, is itself an error estimator of accuracy
O(t
2
) for the time discretization error for that time step. Explicit evaluation
of L
n
would not be expensive compared with the computer time necessary for
the implicit matrix solution, but it does involve coding storage penalties and
complexities (storage of old values of boundary conditions, source terms, and
so on). An economical and elegant approach is to recognize that the L from the
previous implicit step is identical to the L for the present explicit step. Thus,
the L for the present explicit step can be evaluated from the knowledge of the
previous change in f , requiring only the temporary storage of previous solution
arrays. The explicit solution for f at (n +1), F
n+1
, is then obtained by simple
linear extrapolation of previous solutions. For constant t ,
F
n+1
i, j
= 2 f
n
i, j
− f
n−1
i, j
. (31)
For the more general case of variable t ,
F
n+1
i, j
= f
n
i, j
+
t
t
OLD
_
f
n
i, j
− f
n−1
i, j
_
. (32)
The pointwise difference F
n+1
− f
n+1
is then a pointwise temporal error
estimator. Usually, a user would be interested in the maximum over the spatial
domain of the percentage error, so that one would evaluate the error estimator
E
ET
for the single timestep as
E
ET
= 100 · max
i, j
_
abs
_
F
n+1
i, j
− f
n+1
i, j
___
f
RANGE
. (33)
E
ET
is thus calculated as the percent maximum deviation (or L
∞
norm) of the
absolute value of the difference between the new f values f
n+1
, predicted by
the fully implicit algorithm, and F
n+1
, predicted by the explicit algorithm, nor
malized by f
RANGE
, which is the total range of f
n+1
. (In practice, ghostpoint
evaluations with Dirichlet boundary conditions tend to exaggerate the error, so
error estimates and f
RANGE
should be calculated only over interior points.)
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
QUANTIFICATION OF UNCERTAINTY IN CFD 153
Note that the explicit calculation is used only as an error estimator within
a time step of an implicit method, not as the solution algorithm. That is, its
effects do not accumulate. Thus, stability limitations and/or conservation issues
of explicit timestepping as a solution algorithm are irrelevant.
Although the extrapolation procedure is equivalent to explicit timestepping,
the extrapolation cannot be started until there are two time levels. Also, in the
event that the initial conditions are set arbitrarily by the analyst (without setting
initial conditions as a steadystate solution), the initial conditions are likely
to be incompatible with the boundary conditions applied at the ﬁrst timestep.
This means that the change in boundary values during the ﬁrst timestep is
approximately ﬁxed, that is, it does not depend much on the time resolution.
Consequently, the ﬁrst timestep would not provide a meaningful estimate of
∂ f /∂t , and the error estimator would be invalid.
E
ET
can be used readily (internal to the code) as the basis for a solution
adaptive timestepping algorithm, adjusting t so that the error estimate E
ET
is acceptable, either by recalculating the previous timestep (preferred) or by
simply adjusting the next timestep.
Nonlinear Dynamics Solutions
Yee et al (1991) and Lafon &Yee (1992) have given a large number of examples
of nonlinear (chaotic) dynamics solutions in which spurious (grossly erroneous)
solutions for steadystate problems with strongly nonlinear source terms are
obtained for stable implicit algorithms applied beyond the linear stability limit.
Their point is the danger of being misled by such calculations, and the need
for further research to guarantee a priori accuracy. I believe the real lesson
of the examples is simply the need for a posteriori error estimation. In all
but one of the examples, the most cursory timestep convergence study would
quickly reveal the inadequacy of the temporal resolution, with no reasonable
chance of being misled into a nonphysical solution. The one remaining example
(involving a fourthorder RungeKutta integration) could be misleading with a
cursory study but would be revealed in a thorough systematic study. However, it
is worth noting and is perhaps nonintuitive that even steadystate solutions may
require timestep resolution (or equivalently, relaxation parameter resolution)
when signiﬁcant nonlinearities are present.
Cafe Curves
Different ﬁelds of endeavor have different requirements for the variables and
functionals of interest. To generalize too much, engineering (mechanical,
aerospace, chemical) often requires functionals of the solution such as lift coef
ﬁcient, heat transfer rates, mixing rate, and so on. It is much less common that
practical interest is in accuracy of the entire ﬁeld calculation; one example that
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
154 ROACHE
comes to mind is aerooptical propagation (Truman & Lee 1990). However,
meteorological and ocean modelers are usually interested in accuracy of all the
ﬁeld variables. In these cases, it is meaningful to emphasize global accuracy of
the entire computational ﬁeld, and difﬁculty arises in attempting to characterize
global accuracy with just one or a few numbers.
Often L
2
or L
∞
error norms are used as global error indicators. However,
questions arise about whether these error norms are always reliable, and more
signiﬁcantly howthey can be interpreted in any general sense. For example, in a
tidal convergence study (Luettich &Westerink 1995, Westerink et al 1994), L
∞
norms are poor indicators of error because shifting amphidromes, associated
with very high gradient in response amplitude and phase, cause extremely high
localized errors that do not converge smoothly or at the same rate as an L
2
norm.
Furthermore, there is the general problem of interpreting how representative an
L
2
or L
∞
norm is to the response as a whole, or to aspects of the solution of
particular interest.
Luettich &Westerink (1995) introduced the concept of cumulative area frac
tion error (CAFE) curves to present domain errors in a more complete and
meaningful way. These curves plot the fraction of the total domain that ex
ceeds a particular error level (yaxis) against that error level (xaxis). Under
and overprediction are indicated separately. Assuming that under and over
prediction are distributed approximately evenly throughout the domain, median
under and overprediction errors correspond approximately to a cumulative
area fraction of 0.25. For an example of the cumulativeareafraction error
curves, see the grid intercomparisons for tidal ﬂow computations reported by
Westerink et al (1994); see also Westerink & Roache (1995).
The Myth of the Converged Solution
CFD practitioners, especially industrial engineers using generalpurpose com
mercial CFD codes, would like to be able to obtain, at least in principle, a con
verged solution (converged both iteratively and in timespace discretization)
and then be able to conﬁdently postprocess that solution for any engineering
measure of interest. Unfortunately, this is not possible in any general sense.
Again, the problem is the a priori unknown correlation between possible error
measures of valid engineering and scientiﬁc interest. Of course, expert judg
ment (based on experience with a nearby class of problems) can be valid, but
the problem is not soluble in any general sense.
For example, in our experience with dynamic stall and oscillating airfoil
and wing calculations (Salari & Roache 1990, Salari et al 1994), lift was well
converged at the ﬁnest discretization achieved (141 × 55 × 55 in 3D, and
461 × 71 in 2D), drag was more problematical, and moment deﬁnitely was
not converged. If one had performed the gridconvergence study (or other
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
QUANTIFICATION OF UNCERTAINTY IN CFD 155
errorestimation exercise) examining only the lift, one could only infer from
engineering judgment (experience on related problems) whether or not reso
lution was adequate for drag and moment. If a new error measure became of
interest, say position of transition, or location of trailingedge separation, or
second harmonic component of the unsteady pressure coefﬁcient (of interest
for helicopter dynamic stall analysis; McCroskey 1981), the convergence study
would have to be repeated for this error measure, strictly speaking. Calculations
of the branching between symmetric and nonsymmetric ﬂow patterns require
special attention to grid resolution (see Sengupta et al 1995). Rosenfeld (1994)
used grid resolution studies of bluffbody wakes with resolution up to 513 ×513
points, and examined convergence in both the physical and Fourier domains; he
showed that phase velocities of vortices converge much slower than amplitudes,
so that coarser grids may be acceptable for prediction of force coefﬁcients.
Certainly, it is always possible todevise some error measure that is exquisitely
sensitive to discretization error, for example, some highorder statistical corre
lations in turbulent ﬂowcalculations, so that this measure is far from converged
even when other, more benign measures are well converged. A practical exam
ple is a boundarylayer stability calculation. For most engineering applications,
skin friction and wallheat transfer are of principal interest, and these are sen
sitive to the ﬁrst normal derivative of velocity at the wall. However, it is known
from stability theory (Lin 1967) that laminarboundary layer stability is de
pendent on the diffusion of vorticity across the critical layer (the yposition at
which the mean ﬂow speed equals the disturbance wave speed). The principal
component of boundarylayer vorticity magnitude is ∂u/∂y, which means that
the appropriate measure of accuracy is of the term ∂
3
u/∂y
3
, which may be
expected to be more difﬁcult to converge than simply the velocity component
u. Conversely, accurate convergence of pressure may be unnecessary.
Thus, the concept of a converged solution, ascertained to be so independent
of the intended error measure, is a myth.
Error Estimation for Grid Adaptation versus Quantiﬁcation
of Uncertainty
Local error estimators are used for solutionadaptive gridgeneration algo
rithms, and they are usually successful for this purpose (e.g. see Schonauer
et al 1981, Oden et al 1993). However, almost anything intuitive is successful
for adaptation purposes, e.g. minimizing solution curvature or adapting to so
lution gradients (even though solution gradients per se cause no error in most
discretization schemes). In 1D problems (e.g. Salari & Steinberg 1994) or
quasi1D problems (Dwyer et al 1980), the gains in computational efﬁciency
from solution adaptivity of the rtype (redistribution) in structured grids are
very impressive. In real multidimensional problems, gains are usually modest
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
156 ROACHE
(Roache et al 1984, Hall & Zingg 1995) but are somewhat more signiﬁcant for
unstructured grid adaptation (e.g. see Morgan et al 1991, Lohner 1989, Hetu
& Pelletier 1992, Pelletier & Ignat 1995).
But this task of grid adaptation has little connection to the quantiﬁcation of
uncertainty for a ﬁnal calculation with a useful error measure. So the success
of these local error estimators in guiding grid adaptation must not be taken as
demonstration of their efﬁcacy for the quantiﬁcation of uncertainty. Estimates
of local errors usually are not what we want for veriﬁcation of calculations;
we need global errors. By global I do not mean just a global summing up
of local values (as is often used in the FEM literature) but an evaluation that
includes nonlocal effects, that is, one taking into account the fact that errors
are advected, diffused, and so on. For error estimation of useful scientiﬁc or
engineering measures, local estimates are suspect, and their validity as surrogate
estimators for measures of interest must be established anew for each family of
nearby problems.
Internet Archive
An Internet archive of a bibliography for CFD Veriﬁcation and Validation
is maintained by R Barron of the University of Windsor, Ontario, Canada,
and can be accessed at the following address: http://www.lpac.ac.uk/SEL
HPC/Articles/ValCFD.html
Final Remarks
The various approaches to error estimation and quantiﬁcation of uncertainty in
CFDhave their relative merits, involving algebraic simplicity of local estimators
vs the complexity of additional PDE formulations or higherorder accurate so
lutions, singlegrid vs multiplegrid generations, postprocessing vs CFD code
modiﬁcations, and so on. Systematic gridconvergence studies are the most
common, most straightforward and arguably constitute the most reliable tech
nique for the quantiﬁcation of numerical uncertainty. This approach requires no
code modiﬁcations nor algorithm developments but does require multiple grid
generations. Signiﬁcantly, it is not necessary to “double the grid”; noninteger
grid reﬁnement and grid coarsening are economical alternatives.
However, regardless of each approach’s relative merits and of the possibility
for future improvements, it is clear that methods are available now to convinc
ingly assess the numerical uncertainty of CFD calculations. [It is certainly
not necessary, as claimed in a recent SIAM article by Johnson et al (1995), to
analytically and a priori solve the hydrodynamics stability problem in order
to estimate CFD numerical errors!] Journal editors and reviewers can insist
on reasonably thorough assessment without placing impractical demands on
authors, to the improvement of journal quality.
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
QUANTIFICATION OF UNCERTAINTY IN CFD 157
ACKNOWLEDGMENTS
This work was partially supported by Sandia National Laboratories and the
United States Department of Energy under Contract DEAC0476DP00789,
Dr. M Fewell, Technical Monitor.
Visit the Annual Reviews home page at
http://www.annurev.org.
Literature Cited
Aeschliman DP, Oberkampf WL, Blottner FG.
1995. A proposed methodology for compu
tational ﬂuid dynamics code veriﬁcation cal
ibration and validation. Proc. Int. Congr. In
strumentation Aerospace Simulation Facil
ities. Dayton, Ohio: WrightPatterson Air
Force Base
AIAA. 1994. Policy statement on numerical ac
curacy and experimental uncertainty. AIAA J.
32(1):3 (editorial)
ASME Editorial Board. 1994. Journal of Heat
Transfer editorial policy statement on nu
merical accuracy. ASME J. Heat Transfer
116(11):797–98
Babuska I, Strouboulis T, Upadhyay CS. 1994.
A model study of the quality of a posteriori
error estimators for linear elliptic problems.
Error estimation in the interior of patchwise
uniform grid of triangles. Comp. Meth. Appl.
Mech. Engineer. 114:307–78
Blottner FG. 1990. Accurate NavierStokes re
sults for the hypersonic ﬂow over a spher
ical nosetip. AIAA J. Spacecraft Rockets
27(2):113–22
Boehm BW. 1981. Software Engineering Eco
nomics. New York: PrenticeHall
Celik I, Chen CJ, Roache PJ, Scheurer G, eds.
1993. Symposium on Quantiﬁcation of Un
certainty in Computational Fluid Dynamics.
FEDASME, Vol. 158. New York: Am. Soc.
Mech. Eng.
Celik I, Freitas CJ, eds. 1990. Benchmark Test
Cases for Computational Fluid Dynamics.
FEDASME, Vol. 93. New York: Am. Soc.
Mech. Eng.
Celik I, Zhang WM. 1995. Calculation of nu
merical uncertainty using Richardson extrap
olation: application to some simple turbulent
ﬂow calculations. ASME J. Fluids Engineer.
117:439–45
Chang S, Haworth DC. 1995. Kineticenergy
balance based solutionadaptive mesh reﬁne
ment. See Johnson & Hughes 1995, pp. 7–12
Chang S, Haworth DC. 1996. Adaptive grid re
ﬁnement using celllevel and global imbal
ances. Int. J. Num. Meth. Fluids. In press
Claus RW, Vanka SP. 1992. Multigrid calcula
tions of a jet incrossﬂow. J. PropulsionPower
8:185–93
Cosner RR. 1995. CFDValidation requirements
for technology. AIAA 26th Fluid Dynamics
Conf. San Diego, CA
de Vahl Davis G. 1983. Natural convection of air
in a square cavity: a bench mark numerical
solution. Int. J. Num. Meth. Fluids 3(3):249–
64
Dietrich DE, Roache PJ, Marietta MG. 1990.
Convergence studies with the Sandia Ocean
Modeling System. Int. J. Num. Meth. Fluids
11:127–50
Dwyer HA, Kee R, Sanders B. 1980. Adaptive
grid method for problems in ﬂuid mechanics
and heat transfer. AIAA J. 18:1205–12
Ethier CR, Steinmen DA. 1994. Exact fully 3D
NavierStokes solutions for benchmarking.
Int. J. Num. Meth. Fluids 19:369–75
Ewing RE. 1990. A posteriori error estimation.
Comp. Meth. Appl. Mech. Engineer. 82:59–
72
Ferziger JH. 1993. Estimation and reduction of
numerical error. See Celik et al 1993, p. 18
Freitas CJ. 1993a. Editorial policy statement on
the control of numerical accuracy. ASME J.
Fluids Engineer. 115(3):339–40
Freitas CJ, ed. 1993b. The CFD Triathlon:
Three Laminar Flow Simulations by Com
mercial CFD Codes. FEDASME, Vol. 160.
New York: Am. Soc. Mech. Eng.
Freitas CJ. 1994. Response: To the Comments
by Drs. W. Shyy and M. Sindir. ASME J. Flu
ids Engineer. 116(2):198
Freitas CJ. 1995a. Perspective: Selected bench
marks from commercial CFD codes. ASME
J. Fluids Engineer. 117(2):208–18
Freitas CJ. 1995b. The end of justiﬁcation, the
beginning of implementation. ASME J. Flu
ids Engineer. 117(1):9
Gresho PM, Taylor C. 1994. Int. J. Num. Meth.
Fluids 19:iii (editorial)
Hall DJ, Zingg DW. 1995. Viscous Airfoil Com
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
158 ROACHE
putations Using Adaptive Grid Redistribu
tion. AIAA Journal 33 (7):120510
Haworth DC, El Tahry SH, Huebler MS.
1993. A global approach to error estimation
and physical diagnostics in multidimensional
computational ﬂuid dynamics. Int. J. Num.
Meth. Fluids 17:75–97
Hetu JF, Pelletier DH. 1992. Adaptive remesh
ing for viscous incompressible ﬂows. AIAA
J. 30:1986–92
Horgan J. 1995. From complexity to perplexity.
Sci. Am. 272(6):104–9
Johnson C, Rannacher R, Boman M. 1995. Nu
merics and hydrodynamic stability: toward
error control in computational ﬂuid dynam
ics. SIAM J. Num. Anal. 32:1058–79
Johnson RW, Hughes ED, eds. 1995. Quantiﬁ
cation of Uncertainty in Computational Flu
ids Dynamics–1995. FEDASME, Vol. 213.
New York: Am. Soc. Mech. Eng.
Karniadakis GE. 1995. Toward a numerical er
ror bar in CFD. ASME J. Fluids Engineer.
117(1):7–9
Knupp P, Steinburg S. 1994. Fundamentals of
Grid Generation. Boca Raton: CRC
Konikow LF, Bredehoeft JD. 1992. Groundwa
ter models cannot be validated. Adv. Water
Res. 15:75–83
Kuruvila G, Anderson J. 1985. A study of the
effects of numerical dissipation on the calcu
lations of supersonic separated ﬂows. AIAA
Paper 850301
Lafon A, Yee HC. 1992. On the numerical treat
ment of nonlinear source terms in reaction
convection equations. AIAA Paper 920419
Leijnse A, Hassanizadeh SM. 1994. Model def
inition and model validation. Adv. Water Res.
17:197–200
Leonard BP. 1995. Comments on the policy
statement on numerical accuracy. ASME J.
Fluids Engineer. 117(1):5–6
Lin CC. 1967. The Theory of Hydrodynamic
Stability. London: Cambridge Univ. Press
Lohner R. 1989. Adaptive remeshing for tran
sient problems. Comp. Meth. Appl. Mech. En
gineer. 61:323–38
Lotz RD, Thompson BE, Konings CA,
Davoudzadeh F. 1995. Numerical uncertain
ties in transonic ﬂow calculations for airfoils
with blunt trailing edges. Int. J. Num. Meth.
Fluids. In press
Luettich RA, Westerink JJ. 1995. Continen
tal shelf scale convergence studies with a
barotropic tidal model. In Quantitative Skill
Assessment for Coastal Ocean Models, ed.
DR Lynch, AM Davies. Washington: Am.
Geophys. Union. In press
McCroskey WJ. 1981. The Phenomenon of Dy
namic Stall. NASA TM81264
Mehta U. 1989. Computational requirements
for hypersonic ﬂight performance estimates.
AIAA Paper 891670
Mehta U. 1991. Some aspects of uncertainty in
computational ﬂuid dynamics results. ASME
J. Fluids Engineer. 113:538–43
Mehta U. 1995. Guide to credible computa
tional ﬂuid dynamics simulations. AIAA Pa
per 952225
Melnik RE, Siclari MJ, Marconi F, Barger T,
Verhoff A. 1995. An overview of a recent in
dustry effort at CFD code certiﬁcation. AIAA
Paper 952229
Morgan K, Peraire J, Peiro J, Hassan O, Probert
EJ. 1991. Unstructured Grid Methods for
Compressible Flows. In Computational Me
chanics, ed. YK Cheung, JHW Lee, AYY
Leung, 2:1489–97. Rotterdam: Balkema
Nguyen TV, MaclaineCross IL. 1988. Incre
mental pressure drop number in parallel
plate heat exchangers. ASME J. Fluids Eng.
110:93–96
Oberkampf WL. 1994. A proposed framework
for computational ﬂuid dynamics code cali
bration/validation. AIAA Paper 942540
Oberkampf WL, Blottner FG, Aeschliman
DP. 1995. Methodology for Computa
tional Fluid Dynamics Code Veriﬁca
tion/Validation. AIAA Paper 952226
Oden JT, Kennon SR, Tworzydlo WW, Bass
JM, Berry C. 1993. Progress on adaptive h
p ﬁnite element methods for the incompress
ible NavierStokes equations. Comput. Mech.
11:421–32
Oldenburg CM, Pruess K. 1995. Dispersive
transport dynamics in a strongly coupled
groundwaterbrine ﬂow system. Water Re
sources Res. 31(2):289–302
Oreskes N, ShraderFrechette K, Belitz K. 1994.
Veriﬁcation validation and conﬁrmation of
numerical models in the earth sciences. Sci
ence 263:641–46
Organization for Economic Cooperation and
Development. 1988. The International HY
DROCOIN Project Groundwater Hydrology
Modelling Strategies for Performance As
sessment of Nuclear Waste Disposal Level
1: Code Veriﬁcation. OECD Report 71617.
Paris: OEDC
Organization for Economic Cooperation and
Development. 1990. PSACOIN Level 1A In
tercomparison. Paris: OECD
Patankar SV. 1980. Numerical Heat Transfer
and Fluid Flow. New York: McGrawHill
Pelletier D, Ignat L. 1995. On the accuracy
of the grid convergence index and the Zhu
Zienkiewicz error estimator. See Johnson &
Hughes 1995, pp. 31–36
Polya G. 1957. How to Solve It; A New As
pect of Mathematical Method. Princeton, NJ:
Princeton Univ. Press
Putre M. 1994. Verifying analytical models.
ASME Mech. Engineer. 116(11):70–74
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
QUANTIFICATION OF UNCERTAINTY IN CFD 159
Richardson LF. 1908. The approximate arith
metical solution by ﬁnite differences of phys
ical problems involvingdifferential equations
with an application to the stresses in a ma
sonry dam. Trans. R. Soc. London Ser. A,
210:307–57
Richardson LF. 1927. The deferred approach
to the limit. Trans. R. Soc. London Ser. A
226:229–361
Roache PJ. 1972. Computational Fluid Dynam
ics. Albuquerque, NM: Hermosa. Rev. 1976
Roache PJ. 1982. Semidirect/Marching Meth
ods and Elliptic Grid Generation. In Proceed
ings Symposium on the Num. Generation of
Curvilinear Coordinate Systems and Use in
the Numerical Solution of Partial Differen
tial Equations, ed. JFThompson, pp. 727–37.
Amsterdam: NorthHolland
Roache PJ. 1990. Need for control of numerical
accuracy. J. Spacecraft Rockets 27(2):98–102
Roache PJ. 1993a. A method for uniform re
porting of grid reﬁnement studies. See Celik
et al 1993, pp. 109–20
Roache PJ. 1993b. A method for uniform re
porting of grid reﬁnement studies. Proc. 11th
AIAAComput. Fluid Dynam. Conf., Orlando,
FL, 2:1057–58
Roache PJ. 1993c. The SECO code algorithms
for groundwater ﬂow and transport. In Fi
nite elements in ﬂuids: new trends and ap
plications. Proc. VIII Int. Conf. Finite Ele
ments Fluids, ed. K Morgan, E Onate, J Pe
riaux, J Peraire, OC Zienkiewicz, 2:939–48.
Barcelona, Spain: Univ. Politecnica Cataluna
Roache PJ. 1993d. The SECO suite of codes
for site performance assessment. In Proceed
ings 1993 Intl. HighLevel Radioactive Waste
Management Conf. Las Vegas, NV
Roache PJ. 1994a. Perspective: a method for
uniform reporting of grid reﬁnement studies.
ASME J. Fluids Engineer. 116(3):405–13
Roache PJ. 1994b. Response: To the Comments
by Drs. W. Shyy and M. Sindir. ASME J. Flu
ids Engineer. 116(2):198–99
Roache PJ. 1995. Veriﬁcation of codes and cal
culations. AIAA Paper 952224;AIAI J. In
press
Roache PJ, Knupp PM. 1993. Completed
Richardsonextrapolation. Comm. Appl. Num.
Methods 9:365–74
Roache PJ, Salari K. 1990. Weakly compress
ible NavierStokes Solutions with an implicit
approximate factorization code. AIAA Paper
900235
Roache PJ, Ghia K, White F. 1986. Editorial
policy statement on the control of numerical
accuracy. ASME J. Fluids Engineer. 108(1):2
Roache PJ, Knupp PM, Steinberg S, Blaine RL.
1990. Experience with benchmark test cases
for groundwater ﬂow. See Celik & Freitas
1990, pp. 49–56
Roache PJ, Steinberg S, Moeny WM. 1984. In
teractive electric ﬁeld calculations for lasers.
AIAA Paper 841655
Rosenfeld M. 1994. Grid reﬁnement test of
timeperiodic ﬂows over bluff bodies. Comp.
Fluids 23(5):693–709
Salari K, Roache PJ. 1990. The inﬂuence of
sweep on dynamic stall produced by a rapidly
pitching wing. AIAA Paper 900581
Salari K, Steinberg S. 1994. Fluxcorrected
transport in a moving grid. J. Comput.
Physics 111(1):24–32
Salari K, Blaine RL, Economy K, Roache PJ.
1995. Grid resolution studies of radionu
clide transport in fractured porous media. See
Johnson & Hughes 1995, pp. 25–30
Salari K, Roache PJ, Wilcox DC. 1994. Numer
ical Simulation of Dynamic Stall Using the
TwoEquation Kω Turbulence Model, ed. K
Ghia, M Hafez. ASMEFED, 196:1–7. New
York: Am. Soc. Mech. Eng.
Schonauer W, Raith K, Glotz G. 1981. The prin
ciple of difference of difference quotients as a
key to the self adaptive solution of nonlinear
partial differential equations. Comp. Meth.
Appl. Mech. Engineer. 28(3):327–59
Sengupta R, Orkwis PD, Davis SM. 1995. Un
steady solutions transients and local grid res
olution requirements for asymmetric vortex
formation. AIAA Paper 950364
Shih TM, Tan CH, Hwang BC. 1989. Effects of
grid staggering on numerical schemes. Third
Int. J. Num. Meth. Fluids 9(2):193–212
Shirazi SA, Truman CR. 1989. Evaluation of
algebraic turbulence models for PNS predic
tions of supersonic ﬂow past a spherecone.
AIAA J. 27(5):560–68
Shyy W, Sindir M. 1994. Comments on “Policy
statement on the control of numerical accu
racy.” ASME J. Fluids Engineer. 116(2):196–
97
Steinberg S, Roache PJ. 1985. Symbolic manip
ulation and computational ﬂuid dynamics. J.
Comput. Physics 57(2):251–84
Stephens AB, Shubin GR. 1981. Multiple so
lutions and bifurcation of ﬁnite difference
approximations to some steady problems in
ﬂuid dynamics. SIAM J. Sci. Stat. Comput.
2(4):404–15
Strouboulis T, Oden JT. 1990. Aposteriori error
estimation of ﬁnite element approximations
in ﬂuid mechanics. Comp. Meth. Appl. Mech.
Engineer. 78:201–42
Sudicky EA, Frind EO. 1982. Contaminant
transport in fractured porous media: analyti
cal solutions for a systemof parallel fractures.
Water Resources Res. 18(6):1634–42
Truman CR, Lee MJ. 1990. Effects of orga
nized turbulence structures on the phase dis
tortion in a coherent optical beam propagat
ing through a turbulent shear ﬂow. Physics
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
November 28, 1996 12:21 Annual Reviews ROACHE AR2305
160 ROACHE
Fluids A 2(5):851–57
Vanka P. 1994. Response: To the Comments by
Drs. W. Shyy and M. Sindir. ASME J. Fluids
Engineer. 116(2):197–98
Vanka P. 1995. Comment on “Toward a numer
ical error bar in CFD.” ASME J. Fluids Engi
neer. 117(1):9–10
Van Straalen BP, Simpson RB, Stubley GD.
1995. A posteriori error estimation for ﬁnite
volume simulations of ﬂuid ﬂowtransport. In
Proc. Third Annu. Conf. CFD Soc. Canada,
Vol. 1, ed. PAThibault, DMBergeron. Banff,
Alberta
Westerink JJ, Luettich RA, Muccino JC. 1994.
Modeling tides in the western North At
lantic using unstructured graded grids. Tellus
46A:187–99
Westerink JJ, Roache PJ. 1995. Issues in con
vergence studies in geophysical ﬂow compu
tations. Soc. Industr. Appl. Math. In press
Wilcox DC. 1993. Turbulence Modeling for
CFD. La Ca˜ nada, CA: DCW Industries
WIPP PA Dept. 1992. Annual Performance As
sessment for the Waste Isolation Pilot Plant
Dec. 1992 Vol. 1 Preliminary Comparison
with 40 CFR 191 Subpart B Vol. 2: Tech
nical Basis. Albuquerque, NM: Performance
Assess. Dept., Sandia Natl. Lab.
Yee HC, SwebyPK, Grifﬁths DF. 1991. Dynam
ical approach study of spurious steadystate
numerical solutions of nonlinear differential
equations. I. The dynamics of time discretiza
tion and its implications for algorithm devel
opment in computational ﬂuid dynamics. J.
Comput. Physics 97:249–310
Zhu JZ, Zienkiewicz OC. 1990. Superconver
gence recovery technique and a posteriori er
ror estimates. Int. J. Num. Meth. Engineer.
30:1321–39
Zienkiewicz OC, Zhu JZ. 1987. A simple error
estimator and adaptive procedure for practi
cal engineering analysis. Int. J. Num. Meth.
Engineer. 24:337–57
Zienkiewicz OC, Zhu JZ. 1992. The supercon
vergent patch recovery and a posteriori error
estimates. I: The recovery technique. Int. J.
Num. Meth. Engineer. 33:1331–64
Zingg DW. 1993. Grid studies for thinlayer
NavierStokes computations of airfoil ﬂow
ﬁeld. AIAA J. 30(10):2561–64
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
Annual Review of Fluid Mechanics
Volume 29, 1997
CONTENTS
G. I. TAYLOR IN HIS LATER YEARS, J. S. Turner 1
ELECTROHYDRODYNAMICS: The TaylorMelcher Leaky Dielectric
Model, D. A. Saville 27
COREANNULAR FLOWS, D. D. Joseph, R. Bai, K. P. Chen, Y. Y.
Renardy 65
CONVECTION IN MUSHY LAYERS, M. Grae Worster 91
QUANTIFICATION OF UNCERTAINTY IN COMPUTATIONAL
FLUID DYNAMICS, P. J. Roache 123
COMPUTING AERODYNAMICALLY GENERATED NOISE, Valana
L. Wells, Rosemary A. Renaut 161
NONLINEAR BUBBLE DYNAMICS, Z. C. Feng, L. G. Leal 201
PARABOLIZED STABILITY EQUATIONS, Thorwald Herbert 245
QUANTITATIVE FLOW VISUALIZATION IN UNSEEDED FLOWS,
Richard B. Miles and, Walter R. Lempert 285
CONVECTION INTO DOMAINS WITH OPEN BOUNDARIES, T.
Maxworthy 327
FLUID MECHANICS OF SPIN CASTING OF METALS, Paul H.
Steen, Christian Karcher 373
BLOOD FLOW IN ARTERIES, David N. Ku 399
THE PHENOMENOLOGY OF SMALLSCALE TURBULENCE, K. R.
Sreenivasan, R. A. Antonia 435
UNSTRUCTURED GRID TECHNIQUES, D. J. Mavriplis 473
MODERN HELICOPTER AERODYNAMICS, A. T. Conlisk 515
A
n
n
u
.
R
e
v
.
F
l
u
i
d
M
e
c
h
.
1
9
9
7
.
2
9
:
1
2
3

1
6
0
.
D
o
w
n
l
o
a
d
e
d
f
r
o
m
a
r
j
o
u
r
n
a
l
s
.
a
n
n
u
a
l
r
e
v
i
e
w
s
.
o
r
g
b
y
S
t
a
n
f
o
r
d
U
n
i
v
e
r
s
i
t
y
R
o
b
e
r
t
C
r
o
w
n
L
a
w
L
i
b
.
o
n
0
8
/
1
6
/
0
6
.
F
o
r
p
e
r
s
o
n
a
l
u
s
e
o
n
l
y
.
124
ROACHE
here only a posteriori error estimation, being of the opinion that useful a priori estimation is not possible for nontrivial ﬂuid mechanics problems.
Policy Statements on Numerical Uncertainty
In 1986, the editors of the ASME Journal of Fluids Engineering (JFE) saw ﬁt to publish a brief policy statement (Roache et al 1986) requiring at least minimal attention to the quantiﬁcation1 of numerical accuracy. Although the statement seemed innocuous and seemed to address an obvious need, it still met signiﬁcant resistance. (Experience with the implementation of the policy is given in Roache 1990.) Since then, other journals have adopted similar explicit policies (ASME Editorial Board 1994, AIAA 1994, Gresho & Taylor 1994). The JFE has expanded its original policy, including a prohibition of methods with ﬁrstorder spatial accuracy (Freitas 1993a, 1995b; see discussion in Shyy & Shindir 1994, Vanka 1994, Roache 1994b, Freitas 1994; Leonard 1995). An associate editor of the AIAA Journal (W Oberkampf 1995, personal communication) estimates that he needs to cite the journal policy in approximately three quarters of his communications with manuscript authors. The topic has also been the subject of several American Society of Mechanical Engineers (ASME) symposia and resulting series of special publications (Celik & Freitas 1990, Celik et al 1993, Johnson & Hughes 1995), from which the present article takes its title. Also see the related ASME publications on the “CFD Triathlons” (Freitas 1993b, 1995a) and other benchmark comparison exercises, for example, the international HYDROCOIN and PSACOIN projects (OECD 1988, 1990). These exercises give a fair crosssection of present CFD practices in regard to numerical accuracy and generally are far from edifying. Oldenburg & Pruess (1995) give an example of qualitatively different ﬂow revealed by even a cursory grid resolution study that was not discovered by any participants in the HYDROCOIN code intercomparison exercise.
Annu. Rev. Fluid Mech. 1997.29:123160. Downloaded from arjournals.annualreviews.org by Stanford University Robert Crown Law Lib. on 08/16/06. For personal use only.
Veriﬁcation and Validation: Numerical versus Conceptual Modeling
It is useful at the onset to make the important semantic distinction between veriﬁcation and validation. Following Boehm (1981) and Blottner (1990), I adopt the succinct description of veriﬁcation as “solving the equations right,” and of validation as “solving the right equations.” The code author deﬁnes precisely what partial differential equations (PDEs) are being solved and demonstrates convincingly that they are solved correctly—that is, usually with some
1 Other aspects of conﬁdence building in CFD, including broader issues of code qualityassurance, conﬁrmation, calibration, tuning, and certiﬁcation, are found in Mehta (1989, 1991, 1995), Roache et al (1990), Aeschliman et al (1995), Oberkampf (1994), Oberkampf et al (1995), Cosner (1995), Melnik et al (1995), and Roache (1995).
QUANTIFICATION OF UNCERTAINTY IN CFD
125
order of accuracy and always consistently, so that as some measure of discretization (e.g. the mesh increments) approaches zero, the code produces a solution to the continuum PDEs; this is veriﬁcation. Whether or not those equations and that solution bear any relation to a physical problem of interest to the code user is the subject of validation. In a meaningful sense, a code cannot be validated; only a calculation (or range of calculations with a code) can be validated. In my experience, dealing with other than algorithm developers, this is a difﬁcult concept and requires frequent reiteration. Another way to make the distinction (i.e. to get to the idea behind the words, beyond “mere” semantics) is to speak of numerical errors versus conceptual modeling errors. An example of conceptual modeling versus numerical modeling is the assumption of incompressibility. This is clearly a conceptual modeling assumption. Is it the code builder’s fault, or any criticism of the code itself, if the user incorrectly applies it? For example, dynamic stall involves compressibility at a surprisingly low freestream Mach number. Results from an incompressible code may not agree with experiment very well, but we cannot say that the code fails veriﬁcation because it was applied to a compressible ﬂow problem, although we may have some sympathy for the user who is fooled by dynamic stall. But no one would have sympathy for a user who applied an incompressible ﬂow code to a reentry vehicle at Mach 20. In such a case, the lack of agreement with experiment is not a code problem but a modeling problem. The same is true of many practical aspects of applying a CFD code. The model includes more than the code. “Model” includes conceptual modeling assumptions (e.g. incompressibility, symmetry, etc). It also includes data input to the code—e.g. geometry data, which often are not so easy to determine accurately as one might expect, and boundary conditions and initial conditions. Incorrect determination of any of these items can lead to failure of validation of a model, with possibly no criticism of the code. The question of solution uniqueness always arises with nonlinear equations, and its position in the veriﬁcationvalidation distinction is important. If, as stated above, we take veriﬁcation to mean simply that a solution to the continuum PDEs is obtained, then the problem of PDE nonuniqueness is avoided. This is probably being too easy on the code developer in some cases. Physically inadmissable solutions should be eliminated by the code—for example, a shocktube simulator should be required to eliminate expansion shock solutions; but nature abounds with physically nonunique solutions in ﬂuid dynamics—for example, hysteresis of airfoil stall and recovery, and bistable ﬂuid ampliﬁers. If nature cannot decide which solution to produce, we cannot expect more of a code. Note that obtaining “a solution to the continuum PDEs” may involve sorting through multiple numerical solutions for nonlinear problems; for example, see later discussion of Yee et al (1991) and Lafon & Yee (1992). See also
Annu. Rev. Fluid Mech. 1997.29:123160. Downloaded from arjournals.annualreviews.org by Stanford University Robert Crown Law Lib. on 08/16/06. For personal use only.
For personal use only. Fluid Mech.) In a common English thesaurus. but also “whether we can in fact prove a hypothesis false. They make much of supposed intrinsic meaning in the words verify and validate and. . and generally in code qualityassurance (QA). and conﬁrm are all synonyms. we can neither validate nor invalidate Newton’s Law of Gravity. but each term’s technical meaning is deﬁned independent of common use and in a speciﬁc technical context.org by Stanford University Robert Crown Law Lib.” They are talking about physical laws—not just codes but any physical law. (What shall we do? No hazardous waste Annu. but the words are used herein. They come to the remarkable conclusion that it is impossible to verify or validate a numerical model of a natural system. quoting with approval (in their footnote 36) various modern philosophers who question not only whether we can prove any hypothesis true. Downloaded from arjournals. validate. the equally common situation is artiﬁcial stability. validation is very difﬁcult. or of common sense. on 08/16/06. it is asking too much of a code to exactly mimic stability boundaries except in the limit of → 0. Although people usually think of artiﬁcial (i. or of range of applicability. actually think Newton’s law of gravity was proven wrong by Einstein. rather than that Einstein deﬁned the limits of applicability of Newton. an unstable solution is still a valid solution. whose study of Euler solutions indicates that the multiple numerical solutions converge to the same solution as the grid is reﬁned. My impression is that they. agonize over truth. Now most of their concern is with groundwater ﬂow codes. and indeed. In a widely quoted paper that has been recently described as brilliant in an otherwise excellent Scientiﬁc American article (Horgan 1995). The choice of using veriﬁcation or validation was originally arbitrary and is now recommended solely because of common developing use. A related question is the stability of a solution. But Oreskes et al (1994) go much further. But they extend this to all physical sciences. In my opinion.e.29:123160. They clearly have no intuitive concept of error tolerance. for example. as technical terms with more contextspeciﬁc meaning. 1997. Oreskes et al (1994) think that we can ﬁnd the real meaning of a technical term by inquiring about its common meaning. like most lay readers. numerical artifact) instability as a code difﬁculty. Such technical terms are preferably related to common use. a ﬁrstorder timedependent code whose artiﬁcial viscosity damps out disturbances so that accurate calculations of physical ﬂuid instability would require impractically high resolution. it is not a failure of veriﬁcation for a full NavierStokes code to produce steady shearlayer solutions at Reynolds numbers known to be unstable.126 ROACHE especially Stephens & Shubin (1981). Rev. as in a Greek morality play.annualreviews. For example. (I have published articles using the opposite deﬁnition. This is not a universally accepted attitude toward semantics. the terms verify. Speciﬁcally. in geophysics problems.
Numerical errors at boundaries can be ordered in [e. where l is the distance from the region of interest (e. Since the intention in (Karniadakis 1995) is to provide a quantitative breakdown in the sources of numerical error in an error band.29:123160. that outﬂow boundary errors may prove to be ordered not in but in 1/l. . Fluid Mech. no . For personal use only. Indeed. but they do not add new terms. socalled gridgeneration errors (Ferziger 1993) are not separate from discretization errors. Likewise for Karniadakis’s (1995) proposed numerical error bar (see also Vanka 1995).QUANTIFICATION OF UNCERTAINTY IN CFD 127 disposals. on 08/16/06. or that a really bad grid cannot magnify errors. however. bad grids add to discretization error size.annualreviews. If the gridconvergence test is performed. dogs” is not an adequate biological taxonomy. One does not need to separately estimate or band the gridgeneration errors. fauna.g. the boundarylayerlike ∂ p/∂n = 0. 1997. and the errors are shown to reduce as O( 2 ).g. as are dogs. computational domain size. For the veriﬁcation of a code or a calculation. But we see this kind of false taxonomy often in CFD. Several taxonomies of errors given in the literature are inadequate and misleading. Note. but in practical deﬁnitions. or the downstream (outﬂow) boundary location (Roache 1972)]. temporal errors. the various approaches for wall vorticity (Roache 1972)] or possibly nonordered in [e. we are not interested in such worthless semantics and effete philosophizing. For example. applied in the context of engineering and science accuracy.g. but only that these socalled gridgeneration errors do not need to be considered separately from other discretization errors in a gridconvergence test. . so that a boundary error from wall vorticity evaluation need not be considered separately from spatial errors. there are no such things as gridgeneration errors [nor are there “errors associated with coordinate transformations” (Ferziger 1993)]. That taxonomy (Karniadakis 1995) already includes temporal errors and spatial errors and computational domain size errors. no bridges. Inadequate Error Taxonomies Annu. in my opinion. the list “ﬂora. For example. mammals. an airfoil) . and spatial errors. This does not mean that one grid is as good as another. no airplanes. .org by Stanford University Robert Crown Law Lib. which consists of separately estimated numerical errors from boundary conditions. Clearly. for example. so that both ordered and nonordered boundary errors are already counted elsewhere in the taxonomy.) See also Konikow & Bredehoeft (1992) and a rebuttal discussion by Leijnse & Hassanizadeh (1994). then all discretization errors are veriﬁed. Rev. the proposed taxonomy is not only confused but misleading. Not all lists are taxonomies. Mammals are not separate from fauna but are part of it. Ordered errors will tend to zero as the discretization improves. This is clearly a false taxonomy. Downloaded from arjournals.
We have “solved the right equations” only in an intermediate sense of demonstrating that the PNS equations adequately represent the full NS equations.annualreviews. Certainly the equations used are clear. as above. that is. then the distinction is not so clear.” Unfortunately.org by Stanford University Robert Crown Law Lib. Yet to claim validation would be overreaching. Downloaded from arjournals. a simple vortex condition for Euler equations).g. Annu. since we have not demonstrated the adequacy of the turbulence model by comparison with experiment. This agreement is not included in the term veriﬁcation. on 08/16/06. I refer to any measure of discretization (e. [See Zingg’s (1993) data. 1997. number of discrete vortices). that is.g. such “mere semantics” may become of vital interest to code QA when dealing with regulatory agencies such as the EPA or with legal deﬁnitions in a NASA contract. including angle of attack). Rev. since the veriﬁcation of the PNS code has already been completed prior to the NS benchmarking. number of Fourier modes in a spectral solution. Another example of the same semantic failure arises when we consider benchmarking a turbulent boundarylayer code or parabolized NavierStokes (PNS) code against a Reynoldsstressaveraged full NavierStokes (NS) code. The following taxonomy of sources of this additional information provides a framework for the discussion. Fluid Mech. That is. and the code may “solve the equations right” (i. But if the code has some sophisticated outﬂow condition (e. veriﬁcation). Presume that both codes are convincingly veriﬁed. of “solving the right physical equations. shown in Roache (1994a) to be ﬁrstorder in 1/l.] The subject of outﬂow boundary conditions does produce some fuzziness in categorization of Veriﬁcation vs. .128 ROACHE to the outﬂow boundary. The error can be ordered. and therefore can be part of veriﬁcation. Cartesian grid. For personal use only. they correctly solve their respective equations. it is clear that we require some additional information in order to quantitatively estimate the uncertainty or numerical accuracy.g. Then we could say that the agreement has demonstrated that the PNS code is “solving the right equations” in one sense. it justiﬁes the use of parabolic marching equations. but not in the ultimate sense of validation.e. By the word grid. Suppose that the PNS code results agree well with the NS code results for some range of parameters (e. in my opinion. it is up to the user (who is doing the conceptual modeling) to estimate or band the error caused by the position of the outﬂow boundary. yet there exists another benchmark solely from the mathematics (the case with inﬁnite boundary distance) which could be used to justify the outﬂow condition without recourse to physical experiment (which would be associated with validation). Validation. Taxonomy for Additional Information for Error Estimates Once we have produced a CFD solution of the governing partial differential equations. nonorthogonal grid.29:123160.
g. Fluid Mech.2 Grid Coarsening A. most straightforward and arguably constitute the most reliable technique for the quantiﬁcation . is just what you want to guide solution adaptation.QUANTIFICATION OF UNCERTAINTY IN CFD 129 Sources of Additional Information for Error Estimation. no additional grid generation is required. Given a CFD Solution of the Governing PDEs on a Grid A. However.e.1 Higherorder Accuracy Solution(s) B.3 Other Unrelated Grid(s) B.29:123160. Category D. some of which will be justiﬁed and ampliﬁed in later discussion: Categories A and B involve the direct.2 Lowerorder Accuracy Solution(s) C.org by Stanford University Robert Crown Law Lib. needs no additional grid generation. on 08/16/06. Grid Reﬁnement and Coarsening Systematic gridconvergence studies are the most common. Downloaded from arjournals. diffused). no additional CFD code development or modiﬁcations are required. Auxiliary PDE Solutions on the Same Grid D. without advection. the different needs of error estimation for solution adaptation versus quantiﬁcation of uncertainty for a ﬁnal calculation. Additional Solution(s) of the Governing Equations on Other Grids A.1 D.2 D. the error evaluated usually has no direct relation to any error measure of engineering or scientiﬁc interest. Auxiliary Algebraic Evaluations on the Same Grid. 1997. hence. Rev. advected. is cheap.3 D. Surrogate Estimators D. For Category B. Van Straalen et al 1995) does not simply involve a local evaluation of something. However. Additional Solution(s) of the Governing Equations on the Same Grid B. For personal use only. The approach of Category C (e.1 Grid Reﬁnement A.4 Nonconservation of Conservation Variables Nonconservation of Higher Moments ZhuZienkiewicztype Estimators Convergence of Higherorder Quadratures Annu. a simple local evaluation of something. unambiguous evaluation of any error measure of engineering or scientiﬁc interest. The following are brief remarks on this taxonomy. For Category A. The key aspect here is that errors are transported (i.annualreviews. involving algebraic evaluations on the same grid. and uses no signiﬁcant dynamic memory.
g.29:123160. Rev.3) poses an interesting challenge. For noninteger grid reﬁnement (coarsening). see Zingg 1993).org by Stanford University Robert Crown Law Lib. that is. Whether one reﬁnes or coarsens just depends on which grid was calculated ﬁrst. 1994a). Error estimation using other unrelated grids (Category A. grid reﬁnement and coarsening are identical. so the coarse grid solution could be used to estimate the error. For now. The same methods can be converted to verify a code. as well as the usual L 2 and L ∞ norms. A disadvantage of Category A methods is that multiplegrid generation is required. If completely solved solutions are obtained on two grids. Cartesian grids obviously pose no problem.1 (grid reﬁnement). By unrelated grids I mean two or more grids that are overlapping but not simply obtained one from the other. It would seem that the two solutions on unrelated grids would provide the additional information necessary to estimate the uncertainty in either. A Method for Uniform Reporting of GridConvergence Studies: The GridConvergence Index This section presents a summary of the main results of Roache (1993a. that we are using a rigorously veriﬁed code and are now concerned with quantiﬁcation of the uncertainty of a particular calculation using two grid solutions. In order to quantify the uncertainty with systematically reﬁned (coarsened) grids. say grid A is ﬁner than grid B in some regions but coarser in others. to verify (or determine) p. that is. The gridconvergence index (GCI) presents a simple method for uniform reporting of gridconvergence studies without any restriction to integer reﬁnement . See Roache (1993a.130 ROACHE of numerical uncertainty. 1994) for further discussion. Downloaded from arjournals. Unlike the other methods available. the same generating equations and parameters should be used.annualreviews. presumably the ﬁner grid solution would be used. Fluid Mech. such as might be obtained in two steps of an r type (redistribution) solutionadaptive grid.g. this approach can be used to dependably consider the convergence of any quantity of interest.2 (grid coarsening) usually would make more sense. but to my knowledge a method for doing so has not been invented. we are assuming that p is known. For personal use only. we need the convergence rate p to estimate the error. including nonstructured grid reﬁnement (see also Pelletier & Ignat 1995) and structured reﬁnement of nonstructured grids. Annu. For example. but Category A. Code veriﬁcation is discussed in a later section. So for completely solved solutions. using whatever method is preferred (e. the simplest method for grid doubling (halving) is to generate the ﬁnest grid ﬁrst.b. By gridconvergence studies people usually mean Category A. see Knupp & Steinberg 1994) and then obtain the coarser grids by removing every other point (e. 1997. For boundaryﬁtted structured grids. on 08/16/06.
: f [exact] ∼ f 1 + ( f 1 − f 2 )/(r 2 − 1). = (4) It is often stated that Equation 4 is fourthorder accurate if f 1 and f 2 are secondorder accurate. which he achieved by assuming the exclusive use of secondorder centered differences. Richardson Extrapolation Richardson Extrapolation.O. g2 . was ﬁrst used by Richardson in 1910 and later embellished in 1927.29:123160. If uncentered differences are used (e. Using the grid reﬁnement ratio r = h 2 / h 1 . Then the idea is to combine two separate discrete solutions.g. The result is the original statement (Richardson 1927) for h 2 extrapolation: f = f [exact] + h 2 f 1 − h 2 f 2 2 1 h 2 − h 2 + H. and so on. Actually. It is only necessary that Equation 1 be a valid deﬁnition for the order of the discretization.O. and iterated extrapolation.g. Thus. 2 1 (2) where H. The GCI is based on generalized Richardson Extrapolation involving comparison of discrete solutions at two different grid spacings.T. grid doubling). For personal use only. Rev. as known by Richardson. or halving. that is. The discrete solutions f are assumed to have a series representation. Downloaded from arjournals. For inﬁnitely differentiable solutions.) With r = 2. of f = f [exact] + g1 h + g2 h 2 + g3 h 3 + · · · (1) Annu. and obtain a more accurate estimate of f [exact]. nor is the inﬁnite series indicated in Equation 1. this result can be conveniently expressed in terms of a correction to the ﬁnegrid solution f 1 .T. and so on are deﬁned in the continuum and do not depend on any discretization.QUANTIFICATION OF UNCERTAINTY IN CFD 131 (e. the extrapolation may be valid for ﬁnite element solutions. dropping H. these functions are related to all orders to the solution derivatives through the elementary Taylor series expansions. . g1 = 0. these are identical. Fluid Mech. For a secondorder method. we just have a coarse grid and a ﬁne grid.org by Stanford University Robert Crown Law Lib. f 1 and f 2 . also known as h 2 extrapolation. = (3) The most common use of this method is with a grid doubling.O. on 08/16/06. in the grid spacing h. to solve for g2 at the grid points in Equation 1. so as to eliminate the leadingorder error terms in the assumed error expansion. Equation 3 becomes f [exact] ∼ 4/3 f 1 − 1/3 f 2 . 1997. are higherorder terms. this statement is true only if odd powers are absent in the expansion (Equation 1). on two different grids with (uniform) discrete spacings of h 1 (ﬁne grid) and h 2 (coarse grid).annualreviews. that is. The functions g1 . upstream weighting of advection terms). but this is not a necessary assumption for Richardson Extrapolation. (As noted earlier. the deferred approach to the limit. substitute this value into Equation 1.T.
to get an estimate of π with threeﬁgure accuracy. 1927) also considered sixthorder extrapolation (using three grid solutions to eliminate g2 and g4 ). on 08/16/06. Richardson (1927) showed the power of the method in an elegant example of extrapolating two very crude approximations to a circle. but of the extrapolated correction from Equation 4. the extrapolation magniﬁes machine roundoff errors and incomplete iteration errors (Roache 1972). that is. Downloaded from arjournals. singularities.annualreviews. the method is extremely convenient to use compared to forming and solving direct fourthorder discretizations. For example. for the 1D advectiondiffusion equation with Dirichlet boundary conditions. Fluid Mech. and in his 1927 paper had a speciﬁc approach worked out. .g. therefore. as well as the assumption (often veriﬁed) that the local error order is indicative of the global error order. Rev. Richardson (1910. Fourier coefﬁcients. integer grid reﬁnement ratios r (grid doubling. which would seem to require commonality of the discrete solutions and. which involve more complicated stencils. even in his 1910 paper. Rc < 3 is required (Roache & Knupp 1993). In spite of these caveats. not of the solution values f 2 . staggered grids (then called interpenetrating lattices). In order to use Equation 3. Richardson looked forward to deﬁning a continuum f 2 by higherorder interpolation. Annu. Richardson Extrapolation is commonly applied only to obtaining a higherorder estimate on the coarse grid with h 2 = 2h 1 . Similarly. the h 2 extrapolation is thirdorder accurate. Also. This assumption may not be valid for coarse grids.29:123160. Ferziger (1993) alludes to this approach with less detail but more generality. However.org by Stanford University Robert Crown Law Lib. 1997. and other noncalculus problems. not fourth. Although Richardson Extrapolation is most commonly applied to grid doubling and is often stated to be applicable only to integer grid reﬁnement (e. this is not required. The extrapolation must be used with considerable caution. wider bandwidth matrices. tripling. rapid oscillations and the 2h wavelength limit. Conte & DeBoor 1965). As a practical limitation. even extrapolations based on centered differences do not display the anticipated fourthorder accuracy until the cell Reynolds number Rc is reduced. statistical problems. since it involves the additional assumption of monotone truncationerror convergence in the mesh spacing h. The usual assumptions of smoothness apply. by secondorder interpolation of 1/3( f 1 − f 2 ). parabolic and elliptic equations. special considerations for nearboundary points and nonDirichlet boundary conditions. but Roache & Knupp (1993) show how to obtain fourthorder accuracy on all ﬁnegrid points by simple secondorder interpolation. a priori error estimates. For personal use only. integral equations. it is necessary to have values of f 1 and f 2 at the same points. and so on). namely an inscribed square and an inscribed hexagon.132 ROACHE even if these are secondorder accurate (3point upstream). The use of simple secondorder interpolation avoids complexities with nonuniform grids and nearboundary points.
If Richardson Extrapolation is applied to produce (say) fourthorder accurate grid values. one could in principle calculate a fourthorder accurate functional like C L from the grid values. Fluid Mech. that the two approaches yield different answers. this is a major attraction of Richardson Extrapolation compared to using fourthorder accurate stencils solved either directly or by deferred corrections. (Note. experimental agreement was better with coarsegrid calculations than with ﬁne! Also.g. although both are fourthorder accurate if done properly. Zingg’s work demonstrates the necessity of gridconvergence testing even when experimental data are available. Rev. but it would require careful implementation of fourthorder accurate quadratures. lift coefﬁcient C L for an aerodynamics problem or integrated discharge for a groundwater ﬂow problem). with an experimental determination rather than an assumption of the local order of convergence. In four of seven cases. The method is in fact oblivious to the equations being discretized and to the dimensionality of the problem and can easily be applied as a postprocessor (Roache 1982) to solutions on two grids with no reference to the codes. however. as long as the original solutions are indeed secondorder accurate. Such an application was given by the present author in Roache (1982). second. .annualreviews. 1997. with the error being ﬁrst order in the inverse of distance to the boundary (Roache 1994a). his data indicate that Richardson Extrapolation can be applied to the estimation of farﬁeld boundary errors.QUANTIFICATION OF UNCERTAINTY IN CFD 133 additional stability analyses. provided that consistent or higherorder methods are used in the evaluations (e. Blottner (1990) has used the same procedure to estimate effects of artiﬁcial dissipation terms in hypersonic ﬂow calculations.or higherorder quadratures for lift) as well as the basic assumption that the order of the method applies globally as well as locally. obviously being an error estimator (although it does not provide a true bound on the error except possibly for certain trivial problems). An important aspect of Richardson Extrapolation is that it applies not only to pointbypoint solution values but also to solution functionals (e. or governing equations that produced the solutions. Zingg (1993) applied the Richardson error estimator to airfoil lift and drag calculations in bodyﬁtted grids. See Nguyen & MaclaineCross (1988) for application to heat exchanger pressure drop coefﬁcients. requiring only secondorder quadratures. algorithms. Indeed. Downloaded from arjournals. especially in nonorthogonal coordinates that generate crossderivative terms and generally complicated equations. It was used very carefully.) The difference between the secondorder solution and the extrapolated fourthorder solution is itself a useful diagnostic tool. For personal use only.org by Stanford University Robert Crown Law Lib.29:123160.) Annu. I mean a solution obtained with a secondorder accurate method applied in its asymptotic range. by de Vahl Davis (1983) in his classic benchmark study of a model freeconvection problem. on 08/16/06.g. and so on. It is much simpler to apply the extrapolation directly to the C L s obtained in each grid. (By this somewhat abusive but common terminology.
although low level. and p = formal order of accuracy of the algorithm. The extrapolation can introduce noise to the solution.g. the GCI based on the generalized theory of Richardson Extrapolation should be used to uniformly report gridconvergence tests. since that decision involves these considerations and possibly others. Downloaded from arjournals. f 1 . (6) E 2 se = 1−rp where ε = f2 − f1. and is deﬁned as ε f ine E1 = . Rev. 256) pointed out the more fundamental problem that the extrapolated solution is “no longer internally consistent. regardless of whether Richardson Extrapolation is used to improve the solution. (5) 1−rp while a coarsegrid Richardson error estimator approximates the error in a coarsegrid solution. lack of conservation property) source terms into the radionuclide transport equation. it is not advocated (Roache 1993a. For example. 1994a) that Richardson Extrapolation necessarily be used to improve the reported solution. the values of all the variables do not satisfy a system of ﬁnite difference approximations. Fluid Mech. may decrease the accuracy of the solution higher derivatives. which.” It is also noteworthy that Richardson (1927) pointed out that the accuracy of the extrapolation does not apply to arbitrarily high derivatives of the solution. f 1 = a ﬁnegrid numerical solution obtained with grid spacing h 1 . f 2 = a coarsegrid numerical solution obtained with grid spacing h 2 . if it were used on the groundwater ﬂow simulations for the Waste Isolation Pilot Plant (WIPP PA Dept. and is deﬁned as r pε coar . f 1 . A ﬁnegrid Richardson error estimator approximates the error in a ﬁnegrid solution. 1997.e. by comparing this solution to that of a coarse grid. .annualreviews. it would be more accurate in some norm but would introduce additional nonconservative (i.org by Stanford University Robert Crown Law Lib. (7) Annu. f 2 . on 08/16/06. Thus. de Vahl Davis (1983. by comparing the solution to that of a ﬁne grid. p.134 ROACHE A signiﬁcant yet overlooked disadvantage of Richardson Extrapolation is that the extrapolated solution generally is not conservative in the sense of maintaining conservation properties (e. r = reﬁnement factor between the coarse and ﬁne grid (r = h 2 / h 1 > 1). For personal use only.b. What is advocated is that.29:123160. This shortcoming could well dictate that Richardson Extrapolation not be used. Roache 1972). 1992). f 2 .
annualreviews. that a grid adequate for a NACA 0012 airfoil could be assumed to be adequate for a NACA 0015 airfoil. (In fact. we would be expected to use the ﬁnegrid solution. where l = 1 generally or l = 2 if centered differences have been used. if we have a ﬁnegrid and a coarsegrid solution. stall characteristics can be quite sensitive to thickness ratio. For example. E 1 is an ordered estimator and is a good approximation when E 1 As noted above. For personal use only. expecting. Thus. Consider a parametric study in which hundreds of variations are to be run. but most engineers would be satisﬁed with one or a few good gridreﬁnement tests. 1997. p = 2). This requirement implies that the leadingorder truncation error term in the error series truly dominates the error. a safety factor is incorporated into these estimators and the GCI is deﬁned for ﬁne and coarse grids as GCI1 f ine = Fs E 1  (9) (10) GCI coar se = Fs E 2  2 Fs > 1 can be interpreted as a safety factor. I recommended (Roache 1993a. this is often not justiﬁed by experience. 3 rotortip designs.29:123160. However. This value also has the advantage of relating any gridconvergence study (any r and p) to one with a grid doubling and a secondorder method (r = 2. Downloaded from arjournals. 6 airfoilthickness ratios. the error band reduces to the best estimate of the error. To account for the uncertainty in these generalized Richardsonbased error estimates due to various factors and to put all gridconvergence studies on the same basis as grid doubling with a secondorder method. for example. 1994a) as 2 A1 = E 1 + O h p+l . I emphasize that the GCIs are not error estimators but are three (or Fs ) times the error estimators. since Fs = 1 gives GCI = E. . E 1 . it is neither necessary nor often desirable to use r = 2. Rev. Ostensibly.) So for the bulk of the stack of calculations. i. Accurate application of these generalized Richardsonbased griderror estimators requires that the observed convergence rate equals the formal convergence rate. analogous to a 50% error band of experimental data. b. 6 Reynolds numbers.e.QUANTIFICATION OF UNCERTAINTY IN CFD 135 The actual error A1 of the ﬁnegrid solution may be expressed (Roache 1993a. for a total of 648 combinations. (8) Annu.org by Stanford University Robert Crown Law Lib. with perhaps 3 Mach numbers. and 2 turbulence models. representing error bands in a loose statistical sense. a practical scenario occurs for which the contrary situation applies. That is.) A scrupulous approach would require a gridreﬁnement study for each case. Fluid Mech. 1994a) a more conservative value of Fs = 3. 1. we use the coarsegrid solution. or grid doubling (halving). on 08/16/06. consider a 3D timedependent study of dynamic stall. so reporting of the above ﬁnegrid evaluation of GCI would apply. (For example.
Physically realistic solutions will be more convincing to the mathematically naive (and therefore have value in the real world of regulatory agencies. then performance of gridconvergence studies or other errorestimation techniques will not faithfully produce quantiﬁcation of the uncertainty. However. by monitoring grid convergence toward a nontrivial exact solution.25 appears to be adequately conservative.136 ROACHE Annu.g. Signiﬁcantly. Fluid Mech. A recent application of the GCI to airfoil calculations is given by Lotz et al (1995). For personal use only. Pelletier & Ignat (1995) have shown that the GCI is applicable. a modest and more palatable value of Fs = 1. but a mistake). at least in a rough sense. If a code has an error (not an ordered error. the same techniques can be used to verify the code separately. Code Veriﬁcation So far. That is. it is now clear that Fs = 3 is overly conservative for scrupulously performed grid convergence studies using three or more grid solutions to experimentally determine the observed order of convergence p (e. My originally recommended value (Roache 1993a. 1994a) of Fs = 3 is conservative and relates the grid convergence study to one with a grid doubling with a secondorder method. Downloaded from arjournals.b. It has been assumed throughout that the code itself has been veriﬁed independently. this article has been concerned with error estimation. or veriﬁcation of a calculation. and we want a GCI for this solution.29:123160. we derive a GCI from Equation 5. 1997. Rev.org by Stanford University Robert Crown Law Lib. but as the correction to the coarsegrid solution f 2 . For many reasons (see below) this is not unduly conservative when only two grids are used in the study. For such highquality studies. and so on). It is always possible to obtain a nontrivial exact solution for this procedure.annualreviews. In this case. at the insistence of journal editors) I still recommend the value Fs = 3 for the sake of uniform reporting and adequate conservatism. Ethier & Steinmen 1994. that is. . we would be using the coarsegrid solution. on 08/16/06. The motivation for using Fs > 1 is that Fs = 1 is analogous to a 50% error band on experimental data. the error estimate changes and must be less optimistic. by specifying the exact solution and adding the appropriate forcing function to the governing PDEs (Steinberg & Roache 1985. to unstructured grid reﬁnement. The necessary condition is that the solution be nontrivial. which demonstrates the power of the method without the need for integer grid reﬁnement and its application with solutionadaptive grids. if necessary. Shih et al 1989. that it have signiﬁcant solution structure to exercise higherderivative calculations. it is not at all necessary that the solution be realistic in any sense. see the papers in Johnson & Hughes 1995). Westerink & Roache 1995). However. contract management. However. not as the correction to the ﬁnegrid solution f 1 . including its order of convergence. for the more common twogrid study (often performed reluctantly. but the code veriﬁcation is just as well accomplished by nonphysical solutions. which is not adequate.
one must average velocity proﬁles from grid . and has proven remarkably sensitive in revealing small and subtle coding errors and usage inconsistencies. and doubling the number of cells requires the location of the single cell representing the source to shift by /2. (For the alternative philosophy.QUANTIFICATION OF UNCERTAINTY IN CFD 137 Annu. B. the cell faces align with the boundaries of the computational domain. retardation. use of a plausible singlegridblock representation for a point source as the grid is reﬁned introduces error in a ﬁnite volume (or blockcentered ﬁnite difference) formulation. Airfoil codes can exhibit the expected secondorder convergence rates for lift and drag but less for moment. Salari et al 1995): A. on 08/16/06.) Additional realistic solutions (such as Sudicky & Frind 1982. 1997. In veriﬁcation tests of a commercial groundwater ﬂow code. I claim that this method of code veriﬁcation via systematic gridconvergence testing (whether or not the GCI is used) is both general and rigorous. ﬁrstorder extrapolation for ghost cell values of only one quantity (aquifer thickness) along one boundary caused the observed convergence to be ﬁrstorder accurate (Roache 1993c.annualreviews. possibly because of approximations involved in applying quasiperiodicity across cutplanes of a Cgrid (K Salari 1995.org by Stanford University Robert Crown Law Lib.). When a 2D compressible driven cavity problem is solved with an even number of vertical grid lines. It is to be expected that the solution accuracy in the neighborhood of the source would be affected. and Aeschliman et al 1995.d). For personal use only. In contaminant transport calculations (advectiondiffusion + decay. In veriﬁcation tests of our SECO FLOW 2D variable density groundwater ﬂow code. But surprisingly. The observed convergence rate of ostensibly secondorderaccurate turbulentboundarylayer codes (Wilcox 1993) can be degraded. the accuracy of timeintegrated discharge across boundaries far from the source was also degraded to ﬁrstorder accuracy (Salari et al 1995). Rev. see Oberkampf 1994. as indicated by the following examples (Westerink & Roache 1995. Fluid Mech. personal communication). Oberkampf et al 1995. a ﬁrstorder error in a single corner cell in a strongly elliptic problem caused the observed convergence to be ﬁrstorder accurate (Roache et al 1990). In this cell conﬁguration.29:123160. D. E. apparently by conditional statements limiting eddy viscosity and deﬁning the boundarylayer edge (DC Wilcox 1995. C. personal communication. corrected) can be used in additional conﬁrmation exercises (Roache 1995). F. Downloaded from arjournals. and matrix diffusion).
Annu. the situation may be even more difﬁcult than this. it is not enough to perform a calculation of a new problem with a veriﬁed code. it is still necessary to perform gridconvergence tests in order to band the numerical error. Note that in Examples C and F apparently ﬁrstorder behavior was obtained with secondorder veriﬁed codes.” that is. code veriﬁcation must be done before the veriﬁcation of a (real) calculation can proceed. this distinction works both ways. If the . the more difﬁcult and vexing problems come from option combinations and conditional differencing. Fluid Mech. Even though the real problem solution may be converging at the same rate (say secondorder) as the code veriﬁcation problem. that is. but it is not universally so. Downloaded from arjournals. the problem in Example C was a subtle conceptual modeling error and in Example F was a notsosubtle postprocessing error.annualreviews. of the size of the error (as opposed to the order of the error). This unrealistic hope is sometimes encouraged by advertising. Especially in the commercial CFD arena. on 08/16/06.org by Stanford University Robert Crown Law Lib. independent of the semantics. but I am at a loss for a clarifying term. The difﬁcult aspects of the codes are not algebraic complexity [in Steinberg & Roache (1985) we convincingly veriﬁed 1800 lines of dense Fortran]. the codes had no coding errors and were applied in the asymptotic range. user expectations are often that the purchase and use of a “really good code” will remove from the user the obligation of “doing his homework. I distinguish between veriﬁcation of codes and veriﬁcation of individual calculations. this fact provides no estimate of accuracy or conﬁdence interval. TVD. hybrid or typedependent differencing. A code may be rigorously veriﬁed to be (say) secondorder accurate. is that use of a veriﬁed code is not enough.) The very important point. 1997. In both cases. Unfortunately. but when applied to a new problem. For personal use only. Clearly. Esoteric Errors in Code Veriﬁcation General CFD or computational physics codes [more general than the simple Poisson equation in nonorthogonal coordinates of Steinberg & Roache (1985)] would be difﬁcult to include in a theorem because of esoteric errors. in that a new calculation may not even exhibit the veriﬁed order of convergence. Code Veriﬁcation versus Veriﬁcation of Calculations In this review.138 ROACHE lines on each side of the centerline rather than use only one. This point is probably well recognized by present readers.29:123160. the straightforward but tedious work of veriﬁcation of calculations via systematic gridconvergence studies. However. Esoteric errors can arise because of nonlinear ﬂux limiters like FCT. and so on (Steinberg & Roache 1985). otherwise. Rev. the convergence will be ﬁrstorder (Roache & Salari 1990). (It would be preferable to have different words for these two veriﬁcation activities.
g. due to competition of truncation error terms for a particular problem (Westerink & Roache 1995). but local values of observed p near the singularity are required (de Vahl Davis 1983). . This situation typically arises when higher grid resolution reveals higher modes of the solution. the observed convergence rate p may differ from the actual asymptotic convergence rate due to failure to achieve the asymptotic range of grid resolution. which often occurs in atmospheric and ocean modeling (Westerink & Roache 1995. Fluid Mech. However. to observe superconvergence. there is a distinction to be made between the formal order of convergence (as indicated by analyses of the discretization). an actual asymptotic convergence rate higher than a formal rate. For these reasons. a method was presented for completing classical Richardson Extrapolation.29:123160. Then the actual asymptotic rate of convergence may differ from a formal rate. Dietrich et al 1990) and could be expected in high Reynolds number DNS (Direct Numerical Simulation) calculations. Further analysis by PM Knupp then showed that. the order of convergence (e. the higherorder terms in the error expansion include the Reynolds number parameter Re. even though the formal rate was fourth order. and that the asymptotic range for fourthorder accuracy occurs only for Re x 3. and not altogether unusual. Note that there may be more than one formal analysis. 1997. due to cancellation (or partial cancellation) of space and time truncation errors. It is also possible. and the observed (or apparent) order of convergence. because of truncation error competition particular to the steady Burgers solution. so that formal order of convergence is not necessarily unique.g. that is. in which case the method of grid reﬁnement still can be used to estimate uncertainty. The completed algorithm was devised to produce a fourthorder solution on the entire ﬁne grid. The classical method combines two secondorder solutions on a coarse grid and a doubled ﬁne grid to produce a fourthorder accurate solution. For example. This observation is evident in the classical situation of (what are generally) ﬁrst. on 08/16/06. Downloaded from arjournals.annualreviews. a tanh Annu. Rev. it appears to be preferable to verify a code with a nonrealistic problem that has nontrivial but regular solution behavior (e. especially for nonlinear problems. but only on the coarse grid. The cause of the discrepancy between an observed convergence rate and the actual asymptotic convergence rate may or may not be revealed by formal analysis. Finally.org by Stanford University Robert Crown Law Lib.QUANTIFICATION OF UNCERTAINTY IN CFD 139 code veriﬁcation problem(s) are in some sense close to the real problem of interest. secondorder) from the code veriﬁcation exercise may be assumed to hold for the real problem. in Roache & Knupp (1993).or secondorder methods for constant velocity advection problems producing the exact answers for unity Courant number (Roache 1972). However. initial gridconvergence studies indicated only a thirdorder convergence rate for the steadystate Burgers equation. For personal use only. the actual asymptotic order of convergence. This is especially obvious if solution singularities are present.
Without an exact solution for the actual problem. This procedure requires evaluation of the error itself. If the ﬁnest grid solution is taken to be the reference value (unfortunately.140 ROACHE function) rather than to use a realistic problem that displays the confusing truncation error competition. f2 − f1 (11) Annu. For personal use only. then the observed order will be accurate only for those grids far from the ﬁnest. p. three grid solutions are necessary to verify (or determine) the observed rate of convergence and thereby allow estimation of the error. following de Vahl Davis (1983. Thus. Blottner (1990) improves on this by estimating the exact value by Richardson Extrapolation (see also Shirazi & Truman 1989). it is necessary to have at least two grid solutions and an a priori knowledge of the convergence rate p. not restricted to constant r . for example. If an exact solution is known or constructed (see above). With subscript 1 indicating the ﬁnest grid in the present notation. this serves to verify a code. Extraction of Convergence Rate from GridConvergence Tests When using gridconvergence studies to estimate the size of the discretization error. it is also desirable to verify the order p for an actual problem. which is of course generally not known. For constant r . which it obviously is not). it is necessary to have at least three grid solutions to extract p.) One . then the order can be extracted directly from three grid solutions. but this procedure is somewhat ambiguous since the order is already needed in order to perform the Richardson Extrapolation. b. Rev. However. plotting the error on log paper and extracting the order from the slope. and the observed order may differ from the formal (theoretical) order. A generalization of this procedure. but constant). Blottner (1990) and others use graphical means.annualreviews. Downloaded from arjournals. if there is any suspicion that the grid resolution is not in the asymptotic range. to have veriﬁed that the code is secondorder accurate ( p = 2). 1994a). Equation 19 (typo corrected) of Roache (1994a) may be used to verify an assumed order p. on 08/16/06. for a variety of reasons (see discussion above and a more complete discussion in Westerink & Roache 1995). p = ln f3 − f2 / ln(r ). since the observed order of convergence depends on achieving the asymptotic range. often called the exact value. and the calculated order approaching the ﬁnest grid will be indeterminate. which is problem dependent.29:123160. 1997. without a need for estimating the exact solution. 254).org by Stanford University Robert Crown Law Lib. (It is not necessary to use the GCI itself. If the grid reﬁnement is performed with constant r (not necessarily r = 2. or from the order veriﬁed for a test case. is possible using the generalized theory of Richardson Extrapolation (Roache 1993a. it is straightforward to extract the order of convergence p from results of a systematic gridconvergence test using at least two grid solutions. Fluid Mech.
but one should allow for observed p < 1. depends only on the assumed deﬁnition of order of the discretization error. and it cannot be used to calculate p directly since p is implicitly present in the GCIs.5 works well. simple substitution iteration with a relaxation factor ω ∼ 0. For personal use only. of course. Rev. Also. f ine (12) Annu. one may predict the next level of grid reﬁnement r ∗ necessary to achieve a target accuracy.g. With ρ = previous iterate for p. Downloaded from arjournals. Equation 12 also requires r to be constant over the three grid set. expressed as a target error estimate E 1 or a target GC I1 called GC I ∗ . behavior far away from asymptotic convergence can be nonmonotone). . For wellbehaved synthetic cases. 1997. on 08/16/06.) If r is not constant during the grid reﬁnement.org by Stanford University Robert Crown Law Lib. the iteration equation is ln(β) (15a) p = ωρ + (1 − ω) ln(r12 ) β= r12 − 1 ε23 . and r 2 is probably not of much interest. With GC I23 being the value from Equation 2 for the previous two grids. and in some cases the observed p < 0 (unfortunately. If α ≈ r . only on c = error/ p and not on the GCI theory itself. then p is the observed order. This is simple for r constant (not necessarily 2 or integer). giving p = ln(ε23 /ε12 )/ ln r. r ∼ 2 will be easier to solve than r ∼ 1 + δ. r∗ = p GCI∗ /GCI23 . Equation 13 is transcendental in p. ρ r23 − 1 ε12 ρ (15b) Note this form of the iteration gives the exact answer in one step for the case of r = 2 and ω = 0. that is. This result can happen even for simple problems at least locally (de Vahl Davis 1983). Fluid Mech.annualreviews.29:123160. Predicting the Required Grid Resolution Once p is known with some conﬁdence. NewtonRaphson) can apply. However. Usual solution techniques (e. (14) (Note that Equation 14 differs slightly from Equation 11 but is equivalent to the order of the approximation involved. The more general procedure (Roache 1995a) is to solve the equation ε12 ε23 p = r12 (13) p p r23 − 1 r12 − 1 for p.QUANTIFICATION OF UNCERTAINTY IN CFD 141 calculates α = GC I12 p f ine GC I23 . (16) This result.
Annu. the wall functions should be viewed as an elaborate nonlinear boundary condition.annualreviews. for large eddy simulations (LES). Claus & Vanka (1992) found that 2. ENO. Rev.29:123160. For example.4 million nodes (256 × 96 × 96) did not demonstrate grid independence of the computed velocity and turbulence ﬁelds of crossﬂow jets. Virtually any grid is in the asymptotic range for a simple Laplace equation. such as length determinations for the BaldwinLomax turbulence model (Wilcox 1993) can easily corrupt secondorder convergence rates. See Kuruvila & Anderson (1985) for discussion of difﬁculties and pitfalls of doing convergence studies with artiﬁcial dissipation .) This includes both explicit artiﬁcial dissipation (in which an identiﬁable additional term is added to the governing PDEs. For turbulence modeling without wall functions (Wilcox 1993.142 ROACHE Special Considerations for Turbulence Modeling Special considerations are required for turbulence modeling and for other ﬁelds with multiple scales. whether treated explicitly or implicitly in the solution algorithm) and implicit dissipation introduced locally via nonlinear ﬂux limiters (FCT. The other important but troublesome numerical methods are those that treat shocklike problems with any kind of artiﬁcial dissipation that depends on the grid size. Finally. (Shocklike problems means either true shocks or simply regions of unresolvable large solution structure involving solution gradients. and so on). on 08/16/06. it is clear that the initial (coarse) grid must get some points into the boundary layer. once from the full NavierStokes terms and once modeled from the LES terms. the grid convergence must not go to zero. curvatures. 1997. It is necessary to get the grid resolution into the asymptotic range in order to do gridconvergence testing. and various combinations. For turbulence modeling with wall functions. For any boundarylayer calculation. just as laminar boundary layers are much more demanding than inviscid ﬂows. the grid should not get into the wall layer (Celik & Zhang 1995). If not addressed. PPM. Fluid Mech. In my interpretation. or else the Reynolds stresses will be counted twice. these terms corrupt the convergence behavior. For personal use only. Special Considerations for Artiﬁcial Dissipation Special considerations are required for any numerical treatment that effectively changes the governing PDEs as the grid is reﬁned. Also. as used in aerodynamic turbulence research and in atmospheric and ocean modeling with subgrid turbulence modeling. Shirazi & Truman 1989) the grid must get some points into the wall layer. Downloaded from arjournals. Here. the gridresolution requirements are much more demanding for turbulent boundary layers. and the gridconvergence exercise should be done from the edge of the wall layer out. Subgrid turbulence modeling was mentioned earlier.org by Stanford University Robert Crown Law Lib. Similarly. the code theoretical performance can be veriﬁed (within a tolerance) for a range of parameters but could fail in another range. TVD. the presence of any switching functions.
typically when Rc < 2. Over the practical range of resolution of interest. on 08/16/06. The development costs of the additional code capability may be minimized by noting several points.) Richardson (1908) again scooped modern errorestimation papers by inventing Category B.QUANTIFICATION OF UNCERTAINTY IN CFD 143 Annu.annualreviews. Downloaded from arjournals. Fluid Mech. error estimation from higherorder solutions on the same grid. lift. The same illdeﬁned behavior is observed for the class of AllenSouthwell methods (Roache 1972). depending on the local cell Reynolds number Rc. Rev. such as full iterative convergence.” (Again. The explicit dissipation approach can be treated correctly and rigorously. For personal use only. Error Estimation from HigherOrder Solutions on the Same Grid Category B. wherein the emphasis is often on code robustness at the sacriﬁce of accuracy. these Category B methods do not require additional grid generation. However. drag). strict conservation. For the technique of error estimation from higherorder solutions on the same grid. The higherorder accuracy solution might be obtained via a new solution of higherorder discretizations using FDM (ﬁnite difference methods).org by Stanford University Robert Crown Law Lib. much the same advantages and limitations apply as with the gridconvergence technique. note the somewhat abusive but common terminology. By adaptively changing the weighting given to ﬁrstorder and secondorder stencils. as in Blottner (1990) and Shirazi & Truman (1989). However. see Celik & Zhang 1995). Unfortunately. as expected. FVM (ﬁnite volume methods). It applies to all point values and functionals (e. unlike gridconvergence tests (Category A). and the error estimate includes nonlinear coupling. hybrid methods in their various forms cause more problems.g. and so on. but only when the resolution is so ﬁne that the hybrid algorithm itself is inoperative. and so on. many restrictions and requirements could apply. and so on). terms in the equations. On the other hand. FEM (ﬁnite element methods). or by deferred corrections. If we were intending to use the higherorder solution itself.1. The artiﬁcial viscosity of ﬁrstorder upstream differencing (Roache 1972) requires no special consideration other than its slow convergence. but the empirically observed order is. if our only use of the . hybrid methods are commonly used in commercial codes. compact differences (again direct or deferred corrections. somewhere vaguely between 1 and 2 (e. x and t errors may be estimated independently or coupled.29:123160. These methods are not commonly used because they require additional code capability.g. the order is illdeﬁned.1 methods estimate the accuracy of a base solution by comparison with “higherorder accuracy solution(s). 1997. these methods by deﬁnition are ultimately secondorder accurate. noting that the difference between a secondorder accurate solution and a fourthorder accurate solution is itself an ordered error estimator.
directional splitting works. Let f 4x denote the solution obtained with fourthorder . if we deﬁne the error of the fourthorder solution E 4 to be E4 = f − f4 and substituting for f from Equation 18. in two dimensions. The point is that error estimation of. we obtain E 4 = f 2 − f 4 + 0( 2 (21) ) (22) for a secondorder error estimator for the fourthorder solution. (20) This is the basic (Richardson 1908) result that the difference between the second. Then. these considerations are not so important. Rev. a secondorder solution using fourth order methods is less demanding numerically than obtaining a fourthorder solution for direct use. and so on.and fourthorder solutions on the same grid is itself a fourthorder error estimate for the secondorder solution (a result so obvious that it hardly deserves to be named a theorem).144 ROACHE Annu.annualreviews. For personal use only.) We use a notation similar to that above for multiple grid solutions: f denotes the exact solution. f 2 the secondorder accurate solution. Fluid Mech. C2x are the coefﬁcients of the Taylor’s theorem expansion for the secondorder solution in the xdirection. Downloaded from arjournals. f = f 2 + C2x x 2 + R2x + C2y y 2 + R2y and f = f 4 + C4x x 4 + R4x + C4y y 4 + R4y . higherorder solution is to estimate the error of the base solution. and so on.29:123160. on 08/16/06. which can be much easier to implement than a fully directional higherorder solution. and f 4 the fourthorder accurate solution. (The following development has not been completed for crossderivative terms. we obtain E 2 = f 4 − f 2 + 0( 4 (17) (18) (19) ). which in any case may require some careful formulation for higherorder stencils and should be veriﬁed rigorously. though not generally recognized. Likewise obvious. and R2x are the remaining terms in the complete secondorder expansion in x. 1997. The somewhat less obvious result is for the directional splitting of the higherorder solutions. Also. for example. Deﬁning the error of the secondorder solution E 2 to be E2 = f − f2 and substituting for f from Equation 18.org by Stanford University Robert Crown Law Lib.
using Equations 23 and 17. (28) Thus the estimate for the secondorder solution can be obtained to fourth order by directionally split fourthorder solutions. With the deﬁnition given in Equation 19. and f 4y denote the solution obtained with fourthorder discretization in y and secondorder in x. The C2x coefﬁcients are unchanged from the previous deﬁnitions. . For personal use only. (27) (26) In comparison with Equation 17. without approximation. in the same way that grid coarsening can be used rather than grid reﬁnement. 1997. (23) f = f 4y + C4y y 4 + R4y + C2x x 2 + R2x .org by Stanford University Robert Crown Law Lib. we would prove the resulting estimator to be secondorder accurate. Fluid Mech. (25) Similarly. Then f = f 4x + C4x x 4 + R4x + C2y y 2 + R2y and Annu.annualreviews. we estimate the difference between f 4y and f 2 using Equations 24 and 17. Error Estimation from LowerOrder Solutions on the Same Grid Certainly a lowerorder solution on the same grid could be used to estimate the uncertainty of the higherorder solution (Category B. (That is.) The last four terms on the righthand side of Equation 27 are O( 4 ). respectively: f 4y − f 2 = f − C4y y 4 − R4y − C2x x 2 − R2x − { f − C2x x 2 − R2x − C2y y 2 − R2y } = C2y y 2 + R2y − C4y y 4 − R4y . we have the error of the secondorder solution E 2 to be estimated as E 2 = f 4x + f 4y − 2 f 2 + O( 4 ). the ﬁrst four terms on the righthand side of Equation 27 are identically equal to f − f 2 . we show that the estimator is fourthorder accurate) f 4x − f 2 = f − C4x x 4 − R4x − C2y y 2 − R2y − { f − C2x x 2 − R2x − C2y y 2 − R2y } = C2x x 2 + R2x − C4x x 4 − R4x .29:123160.2). (24) We now estimate the difference between f 4x and f 2 . respectively (if we dropped the higherorder remainder terms at this point. but by retaining these terms presently.QUANTIFICATION OF UNCERTAINTY IN CFD 145 discretization in x and secondorder in y. but I know of no such applications. Downloaded from arjournals. the remainder terms R2x and R2y are included. Adding Equations 25 and 26 gives f 4x + f 4y − 2 f 2 = C2x x 2 + R2x + C2y y 2 + R2y −C4x x 4 − R4x − C4y y 4 − R4y . on 08/16/06. Rev.
since the domain of the PDEs requires more than pointwise values. (In fact. which involves realnumber system arithmetic. since the selection and construction of the interelement basis functions is an essential part of the FEM methodology. because somewhere in the development. on 08/16/06. The philosophy of FEM. They invariably conceptualize the numerical solution as more than pointwise. they conceptualize their numerical solution as a continuum solution and. discrete values of the ﬂow variables (as well as related solution functionals. In the Category C methods. That is. Thus. the term residual ordinarily refers to the algebraic residual. the numerical solution consists of pointwise.29:123160. 1997. by evaluating the (algebraic) residual. it makes no sense to FDM or FVM people to speak about estimating the discretization error of a PDE solution. I believe that this difference in attitude and semantics can be traced to the historical accident that FDM practice grew up with iterative solvers. Fluid Mech. the cruder the discretization. obtained by numerical quadratures of the discrete values. When an FEM author talks about residuals. additional approximations are required. In the various ﬂavors of this approach. To most FDM and FVM practitioners. which may be made arbitrarily small no matter how crude and inaccurate the discretization may be. such as drag coefﬁcients and so on. (Incidentally.” There is less than meets the eye in this distinction. the leftovers from the numerical solutions of the discrete algebraic equations. are led to consider the (differential) residual resulting from passing this solution function through the PDE operator.) FEM practitioners have a different viewpoint. thence. but Annu. Rev. . and so on. residuals of the solution are evaluated. is usually distinct from FVM and FDM in this regard. or the underlying conceptual model of FEM practitioners. there can be no exact evaluation of the error of the discrete pointwise solution by passing it through the continuum operator(s) of the governing PDEs. he or she refers to what I distinguish here as differential residuals. these residuals are used as a source term to a linearized error equation that mimics the governing equations. much is made of the distinction between “exact error residual equations” and “approximate error residual equations. Downloaded from arjournals.annualreviews. diffused. that will result from incomplete iterative solution or from accumulated roundoff errors in a direct matrix solver.) To FDM and FVM people. the errors are advected.) Note the major distinction between these approaches: The second (differential) residual would be nonzero even if the computer used true realnumber arithmetic. Indeed. whereas FEM practice grew up with direct solvers. the easier it is to reduce this algebraic residual. often leading to disconnects in communications. For personal use only. Therefore.146 ROACHE Residual Evaluation and Transport In one type of Category C method (auxiliary PDE solutions on the same grid) and in Category D methods as well.org by Stanford University Robert Crown Law Lib.
Rev. when these FEM solutions are graphed for presentation of results. it must be based upon a functional form distinct from that used in the discretization that produced the solution. on 08/16/06. First of all. even though the conservation form (Roache 1972) of the equations was used. and often when they are postprocessed for evaluation of solution functionals such as drag coefﬁcients.annualreviews. . on the FDM/FVM side. Once this useful semantic distinction is made. Downloaded from arjournals. This limitation can be sidestepped by using the weak or integral form of the governing equations. we can examine the differences in the approach. Indeed. 1997. The quadratures can use the same algorithms used for FDM solutions or for any pointwise data. Note that each conceptualization of the underlying (subcell) functional form of the solution leads to a discretization. FVM). the variable coefﬁcient diffusion problem can be conceptualized as having smooth property variation or piecewise constant property variation. a parabolic ﬁt leads to standard secondorder centered difference stencils). the solution may not be truly (locally and globally) conservative. Therefore. Annu. For example. Note. and/or energy. that conceptualizing the solution as a continuum function engenders some problems. except for some algebraic garbage caused by incomplete iterative convergence or machine roundoff.QUANTIFICATION OF UNCERTAINTY IN CFD 147 the ﬁrst (algebraic) residual actually can be driven to machine zero in many (not necessarily all) algorithms/codes. although they produce functions. FDM. however. but the difference is not as essential as one might be led to think.29:123160. that is. Also.org by Stanford University Robert Crown Law Lib. they all essentially work for error estimation and. conserving of mass. the differential residual evaluation will be practically zero. cannot be passed through the PDE operator everywhere because they are not smooth at element boundaries.g. If not. it is not necessary to conceptualize the solution as consisting of only the discrete (node or volume) points. the basis functions are not used. the process (a) is somewhat arbitrary. (In groundwater ﬂow and transport problems. there is no reason why FEM approaches to error estimation (either for solution adaptive grids or for quantiﬁcation of uncertainty) cannot be applied to FDM solutions. Fluid Mech. therefore. All of this discussion is preamble to the most interesting aspect of residual evaluation. a simple methodology for deriving a ﬁnite difference method involves ﬁtting polynomials through node point values (e. Although arguments may be made about which functional form to use for the (differential) residual evaluation. For example. and (b) applies to FDM and FVM as well as FEM. if a (differential) residual is to be evaluated for error evaluation. For personal use only. but the discretization is not necessarily unique to the conceptualization. Regardless of the approach (FEM. momentum. Conversely. we note that most (moderateorder) FEM solutions. The practices are indeed different.
(or lower) order stencils to evaluate the error. Again. the error evaluated usually has no direct relation to any error measure of engineering or scientiﬁc interest. or blends (interpolates) the differentials obtained at element (or cell) centers to produce smooth functions over the entire region. For examples of the error transport equation approach. need no additional grid generation. Whether one selects a C2 smooth function to begin with. The major point is that the residual evaluation must involve a discretization rule different from that used in obtaining the solution. To put it another way. and the globalness of the error estimate becomes dependent on the amount and type of iteration. Annu.148 ROACHE the ﬁrst conceptualization typically is held by geophysicists and engineers. This suggests some tolerance in the construction of subcell solutions. In this way. 1997. However. Rev. or puts off such questions to the quadrature rules. it would be like Category D. and the second by geologists. a higherorder stencil is used to obtain a higherorder solution. Downloaded from arjournals. Auxiliary Algebraic Evaluations on the Same Grid. Fluid Mech. Auxiliary PDE Solutions on the Same Grid The procedure for error estimation in Category C (auxiliary PDE solutions on the same grid) then involves selection of an alternate functional form for the intercell (interelement) solution and evaluation of the (differential) residual. Rather. see Ferziger (1993) and Van Straalen et al (1995). the derivatives may not exist at element boundaries. as expected. which is indeed a signiﬁcant distinction.) Nevertheless. This approach is truly global but not as reliable as Category A or B methods. In Category C.29:123160. The approach in which a rigorously iterated solution is replaced with a less completely converged solution is intermediate. In all cases. For personal use only. etc. are cheap. full NavierStokes) but. involving algebraic evaluations on the same grid. will be a less accurate error estimator near separations and so on. just because your pointwise solution agrees with experiment does not necessarily prove the literalness of your underlying conceptualization. both conceptualizations can lead to the identical discretization and therefore identical discrete solution. on 08/16/06. Surrogate Estimators Category D methods. for a single grid. these methods are after all not so distinct from Category B methods. The residuals can then be used as local error estimators (Category D) or transported in an error equation (Category C). the error transport equation will be some linearized version of the governing equations. These measures can be used as a surrogate for measures of engineering and scientiﬁc interest if and . which use higher. is not very signiﬁcant. and use no signiﬁcant dynamic memory. It will be somewhat cheaper to solve than the original ﬂuid dynamics equations (e. the error measure is deﬁned for mathematical convenience (as are the L 2 and L ∞ norms) or by physical intuition (as is kinetic energy conservation). but if the stencil were used only locally to obtain a local residual.g.org by Stanford University Robert Crown Law Lib.annualreviews.
Rev. which probably should be consistent with the algorithm for solving the PDEs (see also discussion of Category D. For personal use only. but this does not appear to be a strict requirement. Annu. Nonconservation of Conservation Variables Category D. but there is an important and useful distinction to be made in regard to Category D. Although the conservation imbalance may not be of direct engineering or scientiﬁc relevance. when the exact solution is not known. as well as other CFD codes. The error estimate then involves numerical evaluation by quadrature for the erroneous loss or gain of mass in the computational domain.29:123160. In an earlier section. including carefully evaluated boundary inﬂow terms and dissipation (Haworth et al 1993. In Chang & Haworth (1995).annualreviews. if there is any suspicion that the grid resolution is not in the asymptotic range. For example. momentum.1 and D. Chang & Haworth 1995. three grid solutions are necessary to verify (or determine) the rate of convergence and thereby estimate the error. it is also used to reliably guide local grid reﬁnement for solutionadaptive grid generation. the satisfaction of mass conservation will usually depend on the degree of strict iterative convergence achieved. I noted that. it . An evaluation of its global and local conservation by quadrature.2 involves evaluation of conservation errors for higherorder moments. fully conservative discretization may be used for the primitive variables of mass. “when the exact solution is not known. Fluid Mech. on 08/16/06. Note that the evaluation of this error depends on the accuracy of the quadrature. 1996). Conservative codes and algorithms are generally preferred.2 methods that are based on conservation errors. 1997.QUANTIFICATION OF UNCERTAINTY IN CFD 149 only if a correlation is established via numerical experimentation over a suite of problems and range of problem parameters.” may seem redundant when the context is a realistic problem. which is our interest here. vorticity. Downloaded from arjournals.4 below).1 (nonconservation of conservation variables) applies only to codes that do not use fully conservative algorithms for conservation variables. Nonconservation of Higher Moments Category D. but this gives no indication of the discretization error of the solution.) Momentum. and internal energy are other possibilities. then gives a surrogate indication of general discretization errors. in many FEM codes. this error will approach zero but only in the limit of → 0 for a nontrivial problem. mass is not identically conserved in most pre1980 boundarylayer codes. However. The qualiﬁer used. If the coded algorithm is consistent. turbulent kinetic energy is typically not identically conserved. (Even for a nominally massconserving full NavierStokes code. but nonconservative codes are seen to offer a readily evaluated and understood (although still surrogate) error measure. In typical turbulent (ReynoldsAveraged) NavierStokes codes. and internal energy.org by Stanford University Robert Crown Law Lib.
Thus. certainly important for structures problems). Although developed for FEM.3 methods (the ZhuZienkiewicztype estimators) were developed and intended primarily to guide grid adaptation. and Ewing et al (1990). See Zhu & Zienkiewicz (1990). For example.150 ROACHE has the advantage that the exact solution of this quantity is known. These methods allow the global energy norm to be well estimated and give good evaluation of local errors (and provide local estimate of stress accuracy. Oden et al (1993). namely zero. one may build up such correlations by experience and arrive at a practical and very inexpensive error estimator. quadrature for pressure and shear force on the surface leads to evaluation of a drag . personal communication).” The ZhuZienkiewicztype methods share the shortcoming of all surrogate estimators for ﬂuid dynamics—other than guidance for grid adaptation. on 08/16/06. Zienkiewicz & Zhu (1987. using the concepts discussed above in the section “Residual Evaluation and Transport. This need for establishing a correlation in error measures is also true if one direct error measure (e. there is little inherent engineering or scientiﬁc interest in the error measure as deﬁned. Babuska et al (1994). Therefore. the ZhuZienkiewicz approach is adaptable to FDM and FVM (D Pelletier. when one is involved in extensive suites of calculations. does a 1% error estimate on C L ensure a 1% (or 5%. and so on) error estimate for C M ? The correlation providing the (fuzzy) answer will be valid only for a restricted range of angle of attack. and other FEM methods that have a similar ﬂavor. Rev. 1997. and they are relatively cheap. one requires only a single grid solution to calculate the error if p is known. Convergence of HigherOrder Quadratures Category D. in an airfoil calculation. Mach number. such as those of Strouboulis & Oden (1990). Reynolds number. ZhuZienkiewiczType Estimators Category D. For example. involving convergence of higherorder quadratures (numerical integration of a known function).org by Stanford University Robert Crown Law Lib. it is necessary to establish a correlation of the ZhuZienkiewicz estimator with an error measure of interest.annualreviews. nor even for a large range of ﬂow parameters for geometrically similar problems.g. in this surrogate error measure. unless the only interest is mathematics for its own sake. these estimators can be used as surrogate error estimators. Fluid Mech. 1992). Downloaded from arjournals.4 methods are the simplest. However. For example. More relevant in the present context of the quantiﬁcation of uncertainty. and only two grid solutions to extract the observed convergence rate p (see Chang & Haworth 1996). Annu.29:123160. This can be accomplished only by expensive numerical experimentation for a given class of problems. a correlation based on numerical experiments for internal combustion engine modeling would be unlikely to provide any guidance for external aerodynamics. obtained by a gridconvergence study) is to be used as a surrogate for another. and airfoil class. For personal use only.
29:123160. these error estimators would be better named simply resolution indicators.org by Stanford University Robert Crown Law Lib. therefore. the difference may be used as a surrogate error estimator. In the current timestep. For coarse grids. secondorder quadrature will give a different answer than fourthorder accurate quadrature. these values will converge as the solution converges and. We write the backward time method for a general equation system in terms of an operator L (not necessarily linear) as n+1 n f i. the solution was advanced from (n − 1) relative to the .e. The method is very cheap to implement because it does not require another implicit matrix solution. Downloaded from arjournals. It is difﬁcult to take seriously an error estimate for pointwise pressures from an incompressible NavierStokes code when the same error estimate would result from the same pointwise values in an M = 20 solution or in Darcy ﬂow in a porous medium. Fluid Mech. In fact. problems with signiﬁcant solution structure). The same philosophy can be used pointwise. Time Accuracy Estimation Estimation of the numerical error of the time discretization can be done in the same manner as the spatial errors. it is relatively straightforward to estimate the temporal error as the calculation evolves. the poverty of the concept is shown by the fact that the error estimate is obtained from a single solution without recourse to the governing PDEs. coefﬁcient C D .QUANTIFICATION OF UNCERTAINTY IN CFD 151 Annu. (29) Generally. the solution will involve an expensive matrix solution for all f n+1 . The timeerror estimator uses the difference between backward and forward time integration. can be (and have been) used as surrogate error estimates. Rev. implemented as an extrapolation. 1997. In the previous timestep. For nontrivial problems (i. these two quadratures will converge to each other. Indeed. j t = L n+1 . on 08/16/06.annualreviews. For personal use only. since the governing PDEs are not invoked. d) for a fully implicit (backward) time differencing method. Consider the following inexpensive temporal error estimator (Oden et al 1993. However. It includes the effects of timedependent boundary conditions and source terms. Roache 1993c. j − f i. In fact. so that the operator L is being evaluated at (n + 1). As the discretization reﬁnes. but simpler methods may also be used. nor even another explicit stencil evaluation. and to build a code with a solutionadaptive time step to control the temporal error to a predetermined level. The simplest approach ultimately reduces to comparing the calculated pointwise value of some variable of interest with the value interpolated between the point’s neighbors. even when only ﬁrstorder time differencing is used. Equation 29 is advancing the solution for f from time level n to time level (n + 1) with increment t using fully implicit (backward) time differencing.
j The pointwise difference F n+1 − f n+1 is then a pointwise temporal error estimator. and F n+1 . so that one would evaluate the error estimator E ET for the single timestep as n+1 n+1 E E T = 100 · max abs Fi.) The difference between the new value of f n+1 predicted by the implicit algorithm and F n+1 predicted by the explicit algorithm. Thus. as in n+1 n Fi. and so on). j f RANGE . predicted by the explicit algorithm. (31) t. source terms. so error estimates and f RANGE should be calculated only over interior points. both of which predictions are O( t) accurate. 1997. is then obtained by simple linear extrapolation of previous solutions. it does not involve a matrix solution of f at all values of i and j simultaneously. j − f i. For constant t. j − f i. but it does involve coding storage penalties and complexities (storage of old values of boundary conditions. j = 2 f i. is itself an error estimator of accuracy O( t 2 ) for the time discretization error for that time step. Downloaded from arjournals. j = f i. The explicit solution for f at (n + 1).org by Stanford University Robert Crown Law Lib. the L for the present explicit step can be evaluated from the knowledge of the previous change in f . (Since the lefthand side involves values of f only at the location (i. j ± 1) the equation is explicit. j i. j) and (i. Fluid Mech. parallel estimate of the values f n+1 with an explicit step. normalized by f RANGE . Explicit evaluation of L n would not be expensive compared with the computer time necessary for the implicit matrix solution. j . j t = Ln (30) Annu. which is the total range of f n+1 . on 08/16/06.152 ROACHE current indexing to n with increment tOLD and the L evaluated at n. An economical and elegant approach is to recognize that the L from the previous implicit step is identical to the L for the present explicit step. (32) For the more general case of variable n+1 n Fi.29:123160. a user would be interested in the maximum over the spatial domain of the percentage error. Usually. requiring only the temporary storage of previous solution arrays. For personal use only. F n+1 . where F n+1 signiﬁes the new value of f predicted by the explicit algorithm. predicted by the fully implicit algorithm.annualreviews. tOLD i. (In practice.) . j − f i. j) but not at the neighboring locations (i ± 1. We could explicitly evaluate L n and make a separate. n+1 n−1 n Fi. (33) E E T is thus calculated as the percent maximum deviation (or L ∞ norm) of the absolute value of the difference between the new f values f n+1 . Rev. j . ghostpoint evaluations with Dirichlet boundary conditions tend to exaggerate the error. j + t n−1 f n − f i. that is.
not as the solution algorithm. Although the extrapolation procedure is equivalent to explicit timestepping. stability limitations and/or conservation issues of explicit timestepping as a solution algorithm are irrelevant. and so on. that is. In all but one of the examples. and the error estimator would be invalid.org by Stanford University Robert Crown Law Lib. the most cursory timestep convergence study would quickly reveal the inadequacy of the temporal resolution.29:123160. I believe the real lesson of the examples is simply the need for a posteriori error estimation. This means that the change in boundary values during the ﬁrst timestep is approximately ﬁxed. the initial conditions are likely to be incompatible with the boundary conditions applied at the ﬁrst timestep. That is. and the need for further research to guarantee a priori accuracy. either by recalculating the previous timestep (preferred) or by simply adjusting the next timestep. the ﬁrst timestep would not provide a meaningful estimate of ∂ f /∂t. However.annualreviews. Rev. on 08/16/06. Cafe Curves Different ﬁelds of endeavor have different requirements for the variables and functionals of interest. its effects do not accumulate. Note that the explicit calculation is used only as an error estimator within a time step of an implicit method. engineering (mechanical. it is worth noting and is perhaps nonintuitive that even steadystate solutions may require timestep resolution (or equivalently. with no reasonable chance of being misled into a nonphysical solution. chemical) often requires functionals of the solution such as lift coefﬁcient. it does not depend much on the time resolution. heat transfer rates. Thus. Downloaded from arjournals. Nonlinear Dynamics Solutions Yee et al (1991) and Lafon & Yee (1992) have given a large number of examples of nonlinear (chaotic) dynamics solutions in which spurious (grossly erroneous) solutions for steadystate problems with strongly nonlinear source terms are obtained for stable implicit algorithms applied beyond the linear stability limit. 1997. The one remaining example (involving a fourthorder RungeKutta integration) could be misleading with a cursory study but would be revealed in a thorough systematic study. adjusting t so that the error estimate E E T is acceptable. mixing rate.QUANTIFICATION OF UNCERTAINTY IN CFD 153 Annu. aerospace. E E T can be used readily (internal to the code) as the basis for a solutionadaptive timestepping algorithm. It is much less common that practical interest is in accuracy of the entire ﬁeld calculation. relaxation parameter resolution) when signiﬁcant nonlinearities are present. Also. the extrapolation cannot be started until there are two time levels. To generalize too much. Consequently. one example that . Fluid Mech. For personal use only. in the event that the initial conditions are set arbitrarily by the analyst (without setting initial conditions as a steadystate solution). Their point is the danger of being misled by such calculations.
In these cases. meteorological and ocean modelers are usually interested in accuracy of all the ﬁeld variables. see also Westerink & Roache (1995). lift was well converged at the ﬁnest discretization achieved (141 × 55 × 55 in 3D. and more signiﬁcantly how they can be interpreted in any general sense. Furthermore.29:123160. Westerink et al 1994). comes to mind is aerooptical propagation (Truman & Lee 1990). or to aspects of the solution of particular interest. Luettich & Westerink (1995) introduced the concept of cumulative area fraction error (CAFE) curves to present domain errors in a more complete and meaningful way. Salari et al 1994). especially industrial engineers using generalpurpose commercial CFD codes. in a tidal convergence study (Luettich & Westerink 1995. drag was more problematical. would like to be able to obtain. Downloaded from arjournals. there is the general problem of interpreting how representative an L 2 or L ∞ norm is to the response as a whole. Underand overprediction are indicated separately. For example. in our experience with dynamic stall and oscillating airfoil and wing calculations (Salari & Roache 1990. but the problem is not soluble in any general sense. Often L 2 or L ∞ error norms are used as global error indicators.annualreviews. and moment deﬁnitely was not converged. median under.and overprediction are distributed approximately evenly throughout the domain. expert judgment (based on experience with a nearby class of problems) can be valid. The Myth of the Converged Solution CFD practitioners. it is meaningful to emphasize global accuracy of the entire computational ﬁeld. Unfortunately. For an example of the cumulativeareafraction error curves.and overprediction errors correspond approximately to a cumulative area fraction of 0. and 461 × 71 in 2D). Rev. If one had performed the gridconvergence study (or other . on 08/16/06. associated with very high gradient in response amplitude and phase. see the grid intercomparisons for tidal ﬂow computations reported by Westerink et al (1994). the problem is the a priori unknown correlation between possible error measures of valid engineering and scientiﬁc interest. cause extremely high localized errors that do not converge smoothly or at the same rate as an L 2 norm. Again. For example. Of course. this is not possible in any general sense. These curves plot the fraction of the total domain that exceeds a particular error level (yaxis) against that error level (xaxis). For personal use only. Fluid Mech. However. 1997. and difﬁculty arises in attempting to characterize global accuracy with just one or a few numbers. questions arise about whether these error norms are always reliable. L ∞ norms are poor indicators of error because shifting amphidromes.org by Stanford University Robert Crown Law Lib.154 ROACHE Annu. at least in principle. Assuming that under. However.25. a converged solution (converged both iteratively and in timespace discretization) and then be able to conﬁdently postprocess that solution for any engineering measure of interest.
say position of transition. However. so that coarser grids may be acceptable for prediction of force coefﬁcients. and they are usually successful for this purpose (e. In real multidimensional problems. it is always possible to devise some error measure that is exquisitely sensitive to discretization error. accurate convergence of pressure may be unnecessary.annualreviews. gains are usually modest . or location of trailingedge separation. skin friction and wallheat transfer are of principal interest.g. the concept of a converged solution. which means that the appropriate measure of accuracy is of the term ∂ 3 u/∂ y 3 .g. Error Estimation for Grid Adaptation versus Quantiﬁcation of Uncertainty Local error estimators are used for solutionadaptive gridgeneration algorithms. Salari & Steinberg 1994) or quasi1D problems (Dwyer et al 1980). Calculations of the branching between symmetric and nonsymmetric ﬂow patterns require special attention to grid resolution (see Sengupta et al 1995). For personal use only. and examined convergence in both the physical and Fourier domains. is a myth. Fluid Mech. some highorder statistical correlations in turbulent ﬂow calculations. McCroskey 1981). Rosenfeld (1994) used grid resolution studies of bluffbody wakes with resolution up to 513 × 513 points. Rev. the convergence study would have to be repeated for this error measure. for example. 1997. it is known from stability theory (Lin 1967) that laminarboundary layer stability is dependent on the diffusion of vorticity across the critical layer (the yposition at which the mean ﬂow speed equals the disturbance wave speed). minimizing solution curvature or adapting to solution gradients (even though solution gradients per se cause no error in most discretization schemes). The principal component of boundarylayer vorticity magnitude is ∂u/∂ y. the gains in computational efﬁciency from solution adaptivity of the r type (redistribution) in structured grids are very impressive. Conversely. A practical example is a boundarylayer stability calculation. Oden et al 1993). However. so that this measure is far from converged even when other. see Schonauer et al 1981. more benign measures are well converged. strictly speaking.g. If a new error measure became of interest. on 08/16/06. which may be expected to be more difﬁcult to converge than simply the velocity component u. almost anything intuitive is successful for adaptation purposes. Downloaded from arjournals. he showed that phase velocities of vortices converge much slower than amplitudes.org by Stanford University Robert Crown Law Lib.29:123160. Certainly. Thus. errorestimation exercise) examining only the lift.QUANTIFICATION OF UNCERTAINTY IN CFD 155 Annu. e. and these are sensitive to the ﬁrst normal derivative of velocity at the wall. or second harmonic component of the unsteady pressure coefﬁcient (of interest for helicopter dynamic stall analysis. ascertained to be so independent of the intended error measure. In 1D problems (e. For most engineering applications. one could only infer from engineering judgment (experience on related problems) whether or not resolution was adequate for drag and moment.
html Final Remarks The various approaches to error estimation and quantiﬁcation of uncertainty in CFD have their relative merits. it is clear that methods are available now to convincingly assess the numerical uncertainty of CFD calculations. as claimed in a recent SIAM article by Johnson et al (1995). Internet Archive An Internet archive of a bibliography for CFD Veriﬁcation and Validation is maintained by R Barron of the University of Windsor. Signiﬁcantly. (Roache et al 1984. postprocessing vs CFD code modiﬁcations. and can be accessed at the following address: http://www. Fluid Mech. on 08/16/06. However. Rev. local estimates are suspect. and their validity as surrogate estimators for measures of interest must be established anew for each family of nearby problems. to analytically and a priori solve the hydrodynamics stability problem in order to estimate CFD numerical errors!] Journal editors and reviewers can insist on reasonably thorough assessment without placing impractical demands on authors.uk/SELHPC/Articles/ValCFD. singlegrid vs multiplegrid generations. Lohner 1989. it is not necessary to “double the grid”.ac. diffused. we need global errors.29:123160. Canada. Pelletier & Ignat 1995). Downloaded from arjournals.g. see Morgan et al 1991. So the success of these local error estimators in guiding grid adaptation must not be taken as demonstration of their efﬁcacy for the quantiﬁcation of uncertainty. involving algebraic simplicity of local estimators vs the complexity of additional PDE formulations or higherorder accurate solutions. and so on.org by Stanford University Robert Crown Law Lib.156 ROACHE Annu. . most straightforward and arguably constitute the most reliable technique for the quantiﬁcation of numerical uncertainty. Systematic gridconvergence studies are the most common. one taking into account the fact that errors are advected. For personal use only. 1997. Estimates of local errors usually are not what we want for veriﬁcation of calculations.lpac. [It is certainly not necessary.annualreviews. Hall & Zingg 1995) but are somewhat more signiﬁcant for unstructured grid adaptation (e. By global I do not mean just a global summing up of local values (as is often used in the FEM literature) but an evaluation that includes nonlocal effects. and so on. that is. to the improvement of journal quality. Hetu & Pelletier 1992. This approach requires no code modiﬁcations nor algorithm developments but does require multiple grid generations. Ontario. But this task of grid adaptation has little connection to the quantiﬁcation of uncertainty for a ﬁnal calculation with a useful error measure. regardless of each approach’s relative merits and of the possibility for future improvements. For error estimation of useful scientiﬁc or engineering measures. noninteger grid reﬁnement and grid coarsening are economical alternatives.
93. Meth. Fluids Engineer. 1981. Vol. Int. Congr. 1993b. ASME J. Convergence studies with the Sandia Ocean Modeling System. Blottner FG. Mech.org by Stanford University Robert Crown Law Lib. Engineer.annualreviews. AIAA J. Meth. Fluids Engineer. Editorial policy statement on the control of numerical accuracy. For personal use only. Natural convection of air in a square cavity: a bench mark numerical solution. 160. Estimation and reduction of numerical error. Rev. The CFD Triathlon: Three Laminar Flow Simulations by Commercial CFD Codes. Haworth DC. 1995a. J. 1994. 1997. 1994. 115(3):339–40 Freitas CJ. J. See Johnson & Hughes 1995. A proposed methodology for computational ﬂuid dynamics code veriﬁcation calibration and validation. W. ASME J. Comp. Int. Marietta MG. Upadhyay CS. ed. eds. Fluids 19:iii (editorial) Hall DJ. AIAA J.annurev. 1995. Fluids Engineer. Fluids. 114:307–78 Blottner FG. Fluids 3(3):249– 64 Dietrich DE. 18:1205–12 Ethier CR. Spacecraft Rockets 27(2):113–22 Boehm BW. 1994. 158. Celik I. Symposium on Quantiﬁcation of Uncertainty in Computational Fluid Dynamics. Oberkampf WL. 1995. Scheurer G. p. Soc. 7–12 Chang S. San Diego. AIAA 26th Fluid Dynamics Conf. Meth. Viscous Airfoil Com . Shyy and M. 1996. FEDASME. 1990. Strouboulis T. Mech. 1995b. Mech. ASME J. Software Engineering Economics. 117(2):208–18 Freitas CJ. Chen CJ. Num. 1990. A posteriori error estimation. 1980. J. Vol. Annu. CFD Validation requirements for technology. Zhang WM. Roache PJ. Num. FEDASME. 117:439–45 Chang S. Literature Cited Aeschliman DP. Ohio: WrightPatterson Air Force Base AIAA. the beginning of implementation. Accurate NavierStokes results for the hypersonic ﬂow over a spherical nosetip. Meth. 1990. Heat Transfer 116(11):797–98 Babuska I. Multigrid calculations of a jet in crossﬂow. Int. Freitas CJ. Int. 1995. 1994. Policy statement on numerical accuracy and experimental uncertainty. 1994. Soc. Fluids Engineer. New York: Am. Celik I.29:123160. M Fewell. Exact fully 3D NavierStokes solutions for benchmarking. Sanders B. A model study of the quality of a posteriori error estimators for linear elliptic problems. Instrumentation Aerospace Simulation Facilities. Proc. Vol. Freitas CJ. Appl. 1995. Fluids Engineer. Vanka SP. Zingg DW. Adaptive grid reﬁnement using celllevel and global imbalances. Num. New York: Am. Meth.org. Error estimation in the interior of patchwise uniform grid of triangles. 1994. Appl. 1993. 1993. Num. AIAA J. Soc. Downloaded from arjournals. Perspective: Selected benchmarks from commercial CFD codes. Meth. ASME J. 117(1):9 Gresho PM. Eng. J. Technical Monitor. 1995. See Celik et al 1993. Kee R. Fluids 11:127–50 Dwyer HA. Eng. pp. Meth. ASME J. Comp. Benchmark Test Cases for Computational Fluid Dynamics. Fluid Mech. Calculation of numerical uncertainty using Richardson extrapolation: application to some simple turbulent ﬂow calculations. Adaptive grid method for problems in ﬂuid mechanics and heat transfer. New York: PrenticeHall Celik I. Dayton. 18 Freitas CJ. Sindir. J. Visit the Annual Reviews home page at http://www. Fluids 19:369–75 Ewing RE. New York: Am. Kineticenergybalance based solutionadaptive mesh reﬁnement. 32(1):3 (editorial) ASME Editorial Board. Haworth DC. on 08/16/06. FEDASME. 1990. J. Roache PJ. 116(2):198 Freitas CJ. 1983. In press Claus RW. Dr. Engineer. 82:59– 72 Ferziger JH. The end of justiﬁcation. Mech. eds. Int. Int. Taylor C. Response: To the Comments by Drs. ASME J. Num. Propulsion Power 8:185–93 Cosner RR. Steinmen DA. Mech. Eng. Journal of Heat Transfer editorial policy statement on numerical accuracy. 1993a. 1992.QUANTIFICATION OF UNCERTAINTY IN CFD 157 ACKNOWLEDGMENTS This work was partially supported by Sandia National Laboratories and the United States Department of Energy under Contract DEAC0476DP00789. CA de Vahl Davis G.
Science 263:641–46 Organization for Economic Cooperation and Development. Computational requirements for hypersonic ﬂight performance estimates. Num. AIAA Paper 850301 Lafon A. 1993. AIAA Paper 942540 Oberkampf WL. 1994. ed. AYY Leung. On the numerical treatment of nonlinear source terms in reactionconvection equations. Eng. Continental shelf scale convergence studies with a barotropic tidal model. 1981. Water Resources Res. OECD Report 71617. Aeschliman DP. Steinburg S. Water Res. 1992. 113:538–43 Mehta U. DR Lynch. Berry C. Vol. Hassan O. Progress on adaptive hp ﬁnite element methods for the incompressible NavierStokes equations.org by Stanford University Robert Crown Law Lib. Fluid Mech. AIAA Paper 952225 Melnik RE. Am. The Theory of Hydrodynamic Stability. Union. Water Res. Mech. Westerink JJ. 1992. Fluids Engineer. Boca Raton: CRC Konikow LF. Some aspects of uncertainty in computational ﬂuid dynamics results. New York: Am. 2:1489–97. Anderson J. Methodology for Computational Fluid Dynamics Code Veriﬁcation/Validation. Adv. Comments on the policy statement on numerical accuracy. ASME J. Press Putre M. Barger T. On the accuracy of the grid convergence index and the ZhuZienkiewicz error estimator. Engineer. Rotterdam: Balkema Nguyen TV. Int. 1995. AIAA Paper 920419 Leijnse A. 1991. Num. Comp. Blottner FG. Int. 30:1986–92 Horgan J. SIAM J. 31–36 Polya G. Pruess K. In press McCroskey WJ. 17:197–200 Leonard BP.annualreviews. From complexity to perplexity. Annu. Rannacher R. AIAA Journal 33 (7):120510 Haworth DC. A global approach to error estimation and physical diagnostics in multidimensional computational ﬂuid dynamics. MaclaineCross IL. Downloaded from arjournals. JHW Lee. 1995. Meth. Adaptive remeshing for transient problems. Incremental pressure drop number in parallelplate heat exchangers. 31(2):289–302 Oreskes N. J. NJ: Princeton Univ. Toward a numerical error bar in CFD. Verhoff A. Konings CA. Paris: OECD Patankar SV. Appl. Hughes ED. Boman M. Bass JM. FEDASME. AIAA Paper 952229 Morgan K. pp. 1989. Karniadakis GE. El Tahry SH. Geophys. Adv. 213. Rev. ShraderFrechette K. ASME J. The Phenomenon of Dynamic Stall. Press Lohner R. 1985. 1995. Veriﬁcation validation and conﬁrmation of numerical models in the earth sciences. 1967. Peiro J. Anal. Yee HC. Washington: Am. Quantiﬁcation of Uncertainty in Computational Fluids Dynamics–1995. 32:1058–79 Johnson RW. 1990. 1995. London: Cambridge Univ. For personal use only. 116(11):70–74 putations Using Adaptive Grid Redistribution. AIAA Paper 952226 Oden JT. . PSACOIN Level 1A Intercomparison. Princeton. Probert EJ. Davoudzadeh F. Marconi F. Mech. Numerics and hydrodynamic stability: toward error control in computational ﬂuid dynamics. 1995. See Johnson & Hughes 1995. NASA TM81264 Mehta U. Soc. on 08/16/06. Adaptive remeshing for viscous incompressible ﬂows. An overview of a recent industry effort at CFD code certiﬁcation. Fluids Engineer. A New Aspect of Mathematical Method. 110:93–96 Oberkampf WL. 117(1):7–9 Knupp P. Pelletier DH. 1991. Numerical uncertainties in transonic ﬂow calculations for airfoils with blunt trailing edges. Kennon SR. In Computational Mechanics. Paris: OEDC Organization for Economic Cooperation and Development. 117(1):5–6 Lin CC. Huebler MS. Thompson BE. 1980. Bredehoeft JD. Siclari MJ. 1988.29:123160. 1994. The International HYDROCOIN Project Groundwater Hydrology Modelling Strategies for Performance Assessment of Nuclear Waste Disposal Level 1: Code Veriﬁcation. Fundamentals of Grid Generation. Verifying analytical models. 1957. Dispersive transport dynamics in a strongly coupled groundwaterbrine ﬂow system. A proposed framework for computational ﬂuid dynamics code calibration/validation. Model definition and model validation. ASME Mech. In Quantitative Skill Assessment for Coastal Ocean Models. ASME J. 1994. AM Davies. AIAA J. Fluids 17:75–97 Hetu JF. 1995. 1995. New York: McGrawHill Pelletier D. ed. Meth. 1994. 1992. 1995. Engineer.158 ROACHE AIAA Paper 891670 Mehta U. 1994. 1993. 272(6):104–9 Johnson C. J. In press Luettich RA. eds. Num. Hassanizadeh SM. Peraire J. Comput. Mech. Fluids Eng. Unstructured Grid Methods for Compressible Flows. 1995. Belitz K. 1988. 15:75–83 Kuruvila G. Groundwater models cannot be validated. 1997. Tworzydlo WW. Ignat L. Fluids. How to Solve It. Meth. ASME J. Guide to credible computational ﬂuid dynamics simulations. Sci. YK Cheung. 61:323–38 Lotz RD. 1989. 1995. 1995. 11:421–32 Oldenburg CM. 1995. A study of the effects of numerical dissipation on the calculations of supersonic separated ﬂows. Fluids Engineer. Numerical Heat Transfer and Fluid Flow.
Glotz G. See Johnson & Hughes 1995. Num. Fluids 23(5):693–709 Salari K. Comp. Comput. 1984. Shyy and M. Fluid Mech.. Albuquerque. The deferred approach to the limit. AIAA Paper 950364 Shih TM. Stat. VIII Int. The principle of difference of difference quotients as a key to the self adaptive solution of nonlinear partial differential equations. 1990. Knupp PM. 1989. Steinberg S. Proc. 27(5):560–68 Shyy W. The SECO code algorithms for groundwater ﬂow and transport. Grid reﬁnement test of timeperiodic ﬂows over bluff bodies. White F. 1986. Comput. 116(2):198–99 Roache PJ. Soc. K Ghia. 1990. ASMEFED. J. Blaine RL. Computational Fluid Dynamics. AIAA Paper 900235 Roache PJ. Fluids Engineer.QUANTIFICATION OF UNCERTAINTY IN CFD Richardson LF. Roache PJ. E Onate. Engineer. Comp. Perspective: a method for uniform reporting of grid reﬁnement studies. A method for uniform reporting of grid reﬁnement studies. ed. Physics Annu. Response: To the Comments by Drs. 116(2):196– 97 Steinberg S. The approximate arithmetical solution by ﬁnite differences of physical problems involving differential equations with an application to the stresses in a masonry dam. Knupp PM. Trans. Unsteady solutions transients and local grid resolution requirements for asymmetric vortex formation. 109–20 Roache PJ. Ghia K. 1997. Finite Elements Fluids. Meth. Roache PJ. 1981. Need for control of numerical accuracy. 1990. Fluids Engineer. Meth. R. Generation of Curvilinear Coordinate Systems and Use in the Numerical Solution of Partial Differential Equations. 1995. Fluxcorrected transport in a moving grid. 196:1–7. London Ser. Soc. on 08/16/06. Contaminant transport in fractured porous media: analytical solutions for a system of parallel fractures. 1993c. 1908. Sci. Comm. Evaluation of algebraic turbulence models for PNS predictions of supersonic ﬂow past a spherecone. Effects of organized turbulence structures on the phase distortion in a coherent optical beam propagating through a turbulent shear ﬂow. Weakly compressible NavierStokes Solutions with an implicit approximate factorization code. Spacecraft Rockets 27(2):98–102 Roache PJ. Third Int. ed. Mech. Blaine RL. AIAA Paper 900581 Salari K. Steinberg S. Mech. Fluids Engineer. ASME J. HighLevel Radioactive Waste Management Conf. Comput. A 226:229–361 Roache PJ. Appl. Proc. Orlando. Editorial policy statement on the control of numerical accuracy. Spain: Univ. M Hafez. OC Zienkiewicz. In Finite elements in ﬂuids: new trends and applications. Downloaded from arjournals. AIAA Paper 841655 Rosenfeld M. Sindir M. Politecnica Cataluna Roache PJ. 28(3):327–59 Sengupta R. 727–37. Davis SM. ASME J. Eng. 2:939–48. NV Roache PJ. Tan CH. J. Steinberg S. R. K Morgan. 18(6):1634–42 Truman CR. See Celik et al 1993. 1994. Roache PJ. Schonauer W. 1990. 2:1057–58 Roache PJ. A method for uniform reporting of grid reﬁnement studies. 1927. 1982.29:123160. London Ser. Fluids Engineer.annualreviews. J Peraire. Veriﬁcation of codes and calculations. Soc. pp. 1993. 11th AIAA Comput. Raith K. A posteriori error estimation of ﬁnite element approximations in ﬂuid mechanics. Completed Richardson extrapolation. Economy K. Numerical Simulation of Dynamic Stall Using the TwoEquation Kω Turbulence Model. . 108(1):2 Roache PJ. Oden JT.AIAI J. AIAA J. J Periaux. 1993a. pp. FL. New York: Am. Amsterdam: NorthHolland Roache PJ. In Proceedings Symposium on the Num. Roache PJ. Comments on “Policy statement on the control of numerical accuracy. 1985. Experience with benchmark test cases for groundwater ﬂow. 25–30 Salari K. ed. 1994. 1994b. Moeny WM. Barcelona. 1995. 1994. Trans. Orkwis PD.org by Stanford University Robert Crown Law Lib. 2(4):404–15 Strouboulis T. W. Meth. Comp. Lee MJ. Conf. Grid resolution studies of radionuclide transport in fractured porous media. The inﬂuence of sweep on dynamic stall produced by a rapidly pitching wing. Conf. Effects of grid staggering on numerical schemes. For personal use only. 1995. Mech. Interactive electric ﬁeld calculations for lasers. 1976 Roache PJ. Rev. 1990. Num. See Celik & Freitas 1990. 1994. The SECO suite of codes for site performance assessment. Salari K. Appl. J. Engineer. pp. Appl. 1989. 116(3):405–13 Roache PJ. Fluids 9(2):193–212 Shirazi SA. pp. 1981. NM: Hermosa. Frind EO. Las Vegas. 1994a. Shubin GR. Multiple solutions and bifurcation of ﬁnite difference approximations to some steady problems in ﬂuid dynamics. 49–56 159 Roache PJ. 1993d. Hwang BC. 1990. Wilcox DC. In Proceedings 1993 Intl. 1982. 210:307–57 Richardson LF. Physics 57(2):251–84 Stephens AB. Fluid Dynam. Water Resources Res. Semidirect/Marching Methods and Elliptic Grid Generation. 1972. 1993b. Sindir. ASME J. Methods 9:365–74 Roache PJ. SIAM J. Rev. Physics 111(1):24–32 Salari K. In press Roache PJ. J. AIAA Paper 952224.” ASME J. A. Symbolic manipulation and computational ﬂuid dynamics. Truman CR. JF Thompson. 78:201–42 Sudicky EA.
” ASME J. I: The recovery technique. Physics 97:249–310 Zhu JZ. Response: To the Comments by Drs. Turbulence Modeling for CFD. Num. Grid studies for thinlayer NavierStokes computations of airfoil ﬂowﬁeld. J. 1991. The dynamics of time discretization and its implications for algorithm development in computational ﬂuid dynamics. DM Bergeron. 1992. Dept. Fluid Mech. Annual Performance Assessment for the Waste Isolation Pilot Plant Dec. Albuquerque. 1997. Int. Soc. 1987. J. ASME J. 116(2):197–98 Vanka P. 1995. Appl. Vol. 1992. La Ca˜ ada. Modeling tides in the western North Atlantic using unstructured graded grids. 1994. Fluids Engineer. Zienkiewicz OC. Alberta Westerink JJ. Grifﬁths DF. Rev. Sweby PK. Stubley GD. 1993. Comment on “Toward a numerical error bar in CFD. A posteriori error estimation for ﬁnite volume simulations of ﬂuid ﬂow transport. 30:1321–39 Zienkiewicz OC.160 ROACHE with 40 CFR 191 Subpart B Vol. 1993. 1 Preliminary Comparison . Meth. CA: DCW Industries n WIPP PA Dept. Fluids Engineer. Industr.. PA Thibault. W. Superconvergence recovery technique and a posteriori error estimates. 1990. Math. Dynamical approach study of spurious steadystate numerical solutions of nonlinear differential equations. Comput. Luettich RA. 1995. AIAA J. ed. Yee HC. J. For personal use only. 30(10):2561–64 Annu. The superconvergent patch recovery and a posteriori error estimates. Int. Zhu JZ. Sandia Natl. 1992 Vol.annualreviews. Fluids A 2(5):851–57 Vanka P. Zhu JZ. 24:337–57 Zienkiewicz OC. Downloaded from arjournals. on 08/16/06. 1995. Canada. Muccino JC. A simple error estimator and adaptive procedure for practical engineering analysis.org by Stanford University Robert Crown Law Lib. NM: Performance Assess. Meth. Banff. Engineer. 1994. I.29:123160. Simpson RB. Roache PJ. Engineer. Int. Meth. Conf. Third Annu. Num. In Proc. Num. Shyy and M. Tellus 46A:187–99 Westerink JJ. 33:1331–64 Zingg DW. 117(1):9–10 Van Straalen BP. CFD Soc. Issues in convergence studies in geophysical ﬂow computations. Sindir. 2: Technical Basis. Engineer. J. In press Wilcox DC. Lab. 1.
Mavriplis MODERN HELICOPTER AERODYNAMICS. Wells. Grae Worster QUANTIFICATION OF UNCERTAINTY IN COMPUTATIONAL FLUID DYNAMICS. Renardy CONVECTION IN MUSHY LAYERS. Miles and. D. Conlisk 285 327 373 399 435 473 515 . K. Antonia UNSTRUCTURED GRID TECHNIQUES. J. J. J.29:123160. S. on 08/16/06. L. K. 1997. C. Y. R. Y.Annual Review of Fluid Mechanics Volume 29. D. Z. M. Saville COREANNULAR FLOWS.annualreviews. A. Fluid Mech. I. Rosemary A. A. Thorwald Herbert QUANTITATIVE FLOW VISUALIZATION IN UNSEEDED FLOWS. Valana L. Feng. P. Walter R. Downloaded from arjournals. G. Leal PARABOLIZED STABILITY EQUATIONS. D. R. Steen. 1 27 65 91 123 161 201 245 COMPUTING AERODYNAMICALLY GENERATED NOISE. 1997 CONTENTS G. Richard B. Ku THE PHENOMENOLOGY OF SMALLSCALE TURBULENCE. P. Rev. T. Maxworthy FLUID MECHANICS OF SPIN CASTING OF METALS. Lempert CONVECTION INTO DOMAINS WITH OPEN BOUNDARIES. A. TAYLOR IN HIS LATER YEARS. Bai. Paul H. Roache Annu. Joseph. Turner ELECTROHYDRODYNAMICS: The TaylorMelcher Leaky Dielectric Model. T. Chen. R. For personal use only. David N. Sreenivasan. Christian Karcher BLOOD FLOW IN ARTERIES.org by Stanford University Robert Crown Law Lib. D. Renaut NONLINEAR BUBBLE DYNAMICS.