You are on page 1of 10

Fred Stern

Comprehensive Approach to
Professor Mechanical Engineering and
Research Engineer
Fellow ASME
Verification and Validation of CFD
Robert V. Wilson Simulations—Part 1:
Assistant Research Engineer
Mem. ASME Methodology and Procedures
Iowa Institute Hydraulic Research,
The University of Iowa,
Part 1 of this two-part paper presents a comprehensive approach to verification and
Iowa City, IA 52242
validation methodology and procedures for CFD simulations from an already developed
CFD code applied without requiring availability of the source code for specified objec-
tives, geometry, conditions, and available benchmark information. Concepts, definitions,
and equations derived for simulation errors and uncertainties provide the overall math-
Hugh W. Coleman ematical framework. Verification is defined as a process for assessing simulation numeri-
Eminent Scholar in Propulsion,
cal uncertainty and, when conditions permit, estimating the sign and magnitude of the
Professor of Mechanical Engineering,
numerical error itself and the uncertainty in that error estimate. The approach for esti-
Propulsion Research Center,
mating errors and uncertainties includes (1) the option of treating the numerical error as
Mechanical and Aerospace Engineering
deterministic or stochastic, (2) the use of generalized Richardson extrapolation for J input
Department,
parameters, and (3) the concept of correction factors based on analytical benchmarks,
University of Alabama in Huntsville,
which provides a quantitative metric to determine proximity of the solutions to the
Huntsville, AL 35899
asymptotic range, accounts for the effects of higher-order terms, and are used for defining
Fellow ASME
and estimating errors and uncertainties. Validation is defined as a process for assessing
simulation modeling uncertainty by using benchmark experimental data and, when con-
ditions permit, estimating the sign and magnitude of the modeling error itself. The ap-
Eric G. Paterson proach properly takes into account the uncertainties in both the simulation and experi-
Associate Research Engineer,
mental data in assessing the level of validation. Interpretation of results of validation
Iowa Institute Hydraulic Research,
efforts both where the numerical error is treated as deterministic and stochastic are
The University of Iowa,
discussed. Part 2 provides an example for RANS simulations for a cargo/container ship
Iowa City, IA 52242
where issues with regard to practical application of the methodology and procedures and
Mem. ASME
interpretation of verification and validation results are discussed.
关DOI: 10.1115/1.1412235兴

1 Introduction phy is strongly influenced by experimental fluid dynamics 共EFD兲


uncertainty analysis 共Coleman and Steele 关7兴兲, which has been
Discussion and methodology for estimating errors and uncer-
standardized. Hopefully, CFD verification and validation proce-
tainties in computational fluid dynamics 共CFD兲 simulations have
dures and methodology can reach a similar level of maturity and
reached a certain level of maturity with recognition of importance
user variability can reach similar low levels, as for EFD. The work
through editorial policies 共Freitas 关1兴兲, increased attention and re-
is part of a larger program 共Rood 关8兴兲 for developing and imple-
cent progress on common terminology 共AIAA, 关2兴兲, advocacy and
menting a strategy for verification and validation of Reynolds-
detailed methodology 共Roache 关3兴兲, and numerous case studies
averaged Navier-Stokes 共RANS兲 ship hydrodynamics CFD codes.
共e.g. 关4兴兲. Progress has been accelerated in response to the urgent
The program includes complementary CFD and EFD towing-tank
need for achieving consensus on concepts, definitions, and useful
investigations and considers errors and uncertainties in both the
methodology and procedures, as CFD is applied to increasingly
simulations and the data in assessing the success of the verifica-
complex geometry and physics and integrated into the engineering
tion and validation efforts. The work also benefited from collabo-
design process. Such consensus is required to realize the goals of
ration with the 21st and 22nd International Towing Tank Resistance
simulation-based design and other uses of CFD such as simulating
Committees 共ITTC 关9,10兴兲. The procedures proposed in this paper
flows for which experiments are difficult 共e.g., full-scale Reynolds
were adopted on an interim basis by the 22nd ITTC and also were
numbers, hypersonic flows, off-design conditions兲. In spite of the
recommended and used at the recent Gothenburg 2000 Workshop
progress and urgency, the various viewpoints have not converged
on CFD in Ship Hydrodynamics 共Larsson et al. 关11兴兲.
and current approaches fall short of providing practical methodol-
The focus is on verification and validation methodology and
ogy and procedures for estimating errors and uncertainties in CFD
procedures for CFD simulations with an already developed CFD
simulations.
code applied without requiring availability of the source code for
The present work provides a pragmatic approach for estimating
specified objectives, geometry, conditions, and available bench-
errors and uncertainties in CFD simulations. Previous work on
mark information. The methodology and procedures were devel-
verification 共Stern et al. 关5兴兲 is extended and put on a more rigor-
oped considering RANS CFD codes, but should be applicable to a
ous foundation and combined with subsequent work on validation fairly broad range of codes such as boundary-element methods
共Coleman and Stern 关6兴兲 thereby providing a comprehensive and certain aspects of large-eddy and direct numerical simula-
framework for overall procedures and methodology. The philoso- tions. The present work differs in many respects from recent lit-
erature. The presentation is relatively succinct with intention for
Contributed by the Fluids Engineering Division for publication in the JOURNAL
OF FLUIDS ENGINEERING. Manuscript received by the Fluids Engineering Division
use for practical applications 共i.e., industrial CFD兲 for which nu-
November 4, 1999; revised manuscript received July 10, 2001. Associate Editor: merical errors and uncertainties cannot be considered negligible or
P. E. Raad. overlooked.

Journal of Fluids Engineering Copyright © 2001 by ASME DECEMBER 2001, Vol. 123 Õ 793

Downloaded 12 Jan 2011 to 59.162.23.4. Redistribution subject to ASME license or copyright; see http://www.asme.org/terms/Terms_Use.cfm
The definitions of errors and uncertainties and verification and ing to the error in ␦ * . When ␦ * is estimated, it can be used to
validation that are used in any approach need to be clearly stated. obtain a corrected value of the variable of interest.
The present and Roache 关3兴 definitions for errors and uncertainties Sources of errors and uncertainties in results from simulations
are consistent with those used for EFD. The AIAA 关2兴 definitions can be divided into two distinct sources: modeling and numerical.
are from an information theory perspective and differ from those Modeling errors and uncertainties are due to assumptions and ap-
used in EFD, but are not contradictory to the present definitions. proximations in the mathematical representation of the physical
The present concepts and definitions for verification and valida- problem 共such as geometry, mathematical equation, coordinate
tion are closely tied to the present definitions of errors and uncer- transformation, boundary conditions, turbulence models, etc.兲 and
tainties and equations derived for simulation errors and uncertain- incorporation of previous data 共such as fluid properties兲 into the
ties thereby providing the overall mathematical framework. The model. Numerical errors and uncertainties are due to numerical
Roache 关3兴 and AIAA 关2兴 definitions are broader, but not contra- solution of the mathematical equations 共such as discretization, ar-
dictory to the present definitions. The present approach includes tificial dissipation, incomplete iterative and grid convergence, lack
both the situations 共1兲 of estimating errors and the uncertainty of of conservation of mass, momentum, and energy, internal and ex-
those estimates and 共2兲 of estimating uncertainties only. Richard- ternal boundary noncontinuity, computer round-off, etc.兲. The
son extrapolation 共RE兲 is used for verification, which is not new; present work assumes that all correlations among errors are zero,
however, the present generalizations for J input parameters and which is doubtless not true in all cases, but the effects are assumed
concept of correction factors based on analytical benchmarks, negligible for the present analyses.
which provides a quantitative metric to determine proximity of the The simulation error ␦ S is defined as the difference between a
solutions to the asymptotic range, accounts for the effects of simulation result S and the truth T. In considering the develop-
higher-order terms, and are used for defining and estimating errors ment and execution of a CFD code, it can be postulated that ␦ S is
and uncertainties constitute a new approach. The use of quantita- comprised of the addition of modeling and numerical errors
tive estimates for errors and the use of uncertainties for those
estimates also constitute a new approach in verification and ␦ S ⫽S⫺T⫽ ␦ SM ⫹ ␦ SN (1)
validation. Support for this postulation is provided by using the model value
Part 1 of this two-part paper presents the verification and vali- M in definitions for modeling and numerical errors. The simula-
dation methodology and procedures. In Section 2, the overall veri- tion modeling error ␦ SM ⫽M ⫺T is defined as the difference be-
fication and validation methodology is presented by providing tween the true T and model M values while the simulation numeri-
concepts, definitions, and equations for the simulation numerical cal error ␦ SN ⫽S⫺M is defined as the difference between the
and modeling errors and uncertainties. In Section 3, detailed veri- simulation S and model M values. The simulation S and model M
fication procedures for estimation of various sub-components of values are obtained by numerical and exact solutions of the con-
the simulation numerical error and uncertainty are given. In Sec- tinuous equations used to model the truth, respectively. Since ex-
tion 4, validation procedures are given including a discussion of act solution of nonlinear equations is seldom possible, approxima-
the interpretation of validation results and use of corrected simu- tions are used to replace the continuous modeled equations with
lation results. Finally, conclusions are provided in Section 5. Part discrete ones that are solved algebraically with a CFD code to
2 provides an example for RANS simulations for a cargo/ yield the simulation value S.
container ship where issues with regard to practical application of The uncertainty equation corresponding to error equation 共1兲 is
the methodology and procedures and interpretation of verification
and validation results are discussed 共Wilson et al. 关12兴兲. Present U 2S ⫽U SM
2
⫹U SN
2
(2)
papers are based on Stern et al. 关13兴, which is sometimes refer- where U S is the uncertainty in the simulation and U SM and U SN
enced for additional details. However, presentation and expanded are the simulation modeling and numerical uncertainties.
discussions of verification procedures and implementation were For certain conditions, the numerical error ␦ SN can be consid-
improved based on nearly two years experience with present ap- ered as
proach, especially through ITTC community and Gothenburg
2000 Workshop on CFD in Ship Hydrodynamics. ␦ SN ⫽ ␦ SN
* ⫹␧ SN (3)
where ␦ SN* is an estimate of the sign and magnitude of ␦ SN and
2 Overall Verification and Validation Methodology ␧ SN is the error in that estimate 共and is estimated as an uncertainty
since only a range bounding its magnitude and not its sign can be
In Section 2.1, the overall verification and validation method- estimated兲. The corrected simulation value S C is defined by
ology is presented by providing key concepts, definitions, and
derivation of equations for the simulation error and uncertainty, as S C ⫽S⫺ ␦ SN
* (4)
sum and root-sum-square 共RSS兲 of simulation numerical and
modeling errors and uncertainties, respectively. The verification with error equation
and validation equations are derived in Sections 2.2 and 2.3, re- ␦ S C ⫽S C ⫺T⫽ ␦ SM ⫹␧ SN (5)
spectively, where subcomponents of the simulation numerical er-
ror are identified and an approach for assessing the simulation The uncertainty equation corresponding to error equation 共5兲 is
modeling uncertainty is presented.
U S2 C ⫽U SM
2
⫹U S2 C N (6)
2.1 Concepts and Definitions. Accuracy indicates the
closeness of agreement between a simulation/experimental value where U S C is the uncertainty in the corrected simulation and U S C N
of a quantity and its true value. Error ␦ is the difference between is the uncertainty estimate for ␧ SN .
a simulation value or an experimental value and the truth. Accu- Debate on verification and validation has included discussion
racy increases as error approaches zero. The true values of on whether errors such as ␦ SN are deterministic or stochastic, and
simulation/experimental quantities are rarely known. Thus, errors thus how they should be treated in uncertainty analysis was un-
must be estimated. An uncertainty U is an estimate of an error clear. In the ‘‘corrected’’ approach given by Eqs. 共3兲–共6兲, a deter-
such that the interval ⫾U contains the true value of ␦ 95 times ministic estimate ␦ SN
* of ␦ SN and consideration of the error ␧ SN in
out of 100. An uncertainty interval thus indicates the range of that estimate are used. The approach is analogous to that in EFD
likely magnitudes of ␦ but no information about its sign. when an asymmetric systematic uncertainty is ‘‘zero-centered’’ by
For simulations, under certain conditions, errors can be esti- inclusion of a model for the systematic error in the data reduction
mated including both sign and magnitude 共referred to as an error equation and then the uncertainty considered is that associated
estimate ␦ * 兲. Then, the uncertainty considered is that correspond- with the model 共Coleman and Steele 关7兴兲. In the ‘‘uncorrected’’

794 Õ Vol. 123, DECEMBER 2001 Transactions of the ASME

Downloaded 12 Jan 2011 to 59.162.23.4. Redistribution subject to ASME license or copyright; see http://www.asme.org/terms/Terms_Use.cfm
approach given by Eqs. 共1兲–共2兲, any particular ␦ SN is considered
as a single realization from some parent population of ␦ SN ’s and
the uncertainty U SN is interpreted accordingly in analogy to the
estimation of uncertainties in EFD 共with a similar argument for
␧ SN and U S C N 兲. Oberkamp and Trucano 关14兴 have criticized Cole-
man and Stern 关6兴 for treating U SN statistically; however, the
present approach is well justified both conceptually and math-
ematically for reasons just given.
The overall CFD verification and validation procedures can be
conveniently grouped in four consecutive steps. The first step is
preparation, which involves selection of the CFD code and speci-
fication of objectives, geometry, conditions, and available bench-
mark information. The objectives might be prediction of certain
variables at certain levels of validation 共e.g., programmatic vali-
dation requirements U reqd 兲. The variables can either be integral
共e.g., resistance兲 or point 共e.g., mean velocities and turbulent Rey-
nolds stresses兲 values and the programmatic validation require-
ments may be different for each variable. The second and third
Fig. 1 Definition of comparison error.
steps are verification and validation, which are described in Sec-
tions 2.2 and 2.3. The fourth step is documentation, which is
detailed presentation of the CFD code 共equations, initial and
boundary conditions, modeling, and numerical methods兲, objec- mark since it is equal, as shown by Eq. 共10兲, to the truth plus
tives, geometry, conditions, verification, validation, and analysis. simulation modeling error and presumable small error ␧ SN in the
2.2 Verification. Verification is defined as a process for as- estimate of the numerical error ␦ SN
* .
sessing simulation numerical uncertainty U SN and, when condi- 2.3 Validation. Validation is defined as a process for assess-
tions permit, estimating the sign and magnitude ␦ SN* of the simu- ing simulation modeling uncertainty U SM by using benchmark
lation numerical error itself and the uncertainty in that error experimental data and, when conditions permit, estimating the
estimate. For many CFD codes, the most important numerical sign and magnitude of the modeling error ␦ SM itself. Thus, the
errors and uncertainties are due to use of iterative solution meth- errors and uncertainties in the experimental data must be consid-
ods and specification of various input parameters such as spatial ered in addition to the numerical errors and uncertainties dis-
and time step sizes and other parameters 共e.g., artificial dissipa- cussed in Section 3. Approaches to estimating experimental un-
tion兲. The errors and uncertainties are highly dependent on the certainties are presented and discussed by Coleman and Steele 关7兴.
specific application 共geometry and conditions兲. The validation methodology of Coleman and Stern 关6兴 which
The errors due to specification of input parameters are decom- properly takes into account the uncertainties in both the simula-
posed into error contributions from iteration number ␦ I , grid size tion and the experimental data is discussed in this section for both
␦ G , time step ␦ T , and other parameters ␦ P , which gives the approaches of treating the numerical error as stochastic and as
following expressions for the simulation numerical error and un- deterministic.
certainty The validation comparison is shown in Fig. 1. The experimen-
J
tally determined r-value of the (X i ,r i ) data point is D and simu-
lated r-value is S. Recall from Eq. 共1兲 that the simulation error ␦ S
␦ SN ⫽ ␦ I ⫹ ␦ G ⫹ ␦ T ⫹ ␦ P ⫽ ␦ I ⫹ 兺␦
j⫽1
j (7) is the difference between S and the truth T. Similarly, the error ␦ D
in the data is the difference between D and the truth T, so setting
J the simulation and experimental truths equal results in
2
U SN ⫽U I2 ⫹U G
2
⫹U T2 ⫹U 2P ⫽U I2 ⫹ 兺U
j⫽1
2
j (8) D⫺ ␦ D ⫽S⫺ ␦ S (13)
The comparison error E is defined as the difference of D and S
Similarly, error estimates ␦ * can be decomposed as
E⫽D⫺S⫽ ␦ D ⫺ ␦ S ⫽ ␦ D ⫺ 共 ␦ SM A ⫹ ␦ S PD ⫹ ␦ SN 兲 (14)
J
with ␦ SM decomposed into the sum of ␦ S PD , error from the use of
␦ SN
* ⫽ ␦ I* ⫹ 兺 ␦*
j⫽1
j (9) previous data such as fluid properties, and ␦ SM A , error from mod-
eling assumptions. Thus E is the resultant of all the errors asso-
which gives the following expressions for the corrected simulation ciated both with the experimental data and with the simulation.
and corrected simulation numerical uncertainty For the approach in which no estimate ␦ SN * of the sign and mag-
nitude of ␦ SN is made, all of these errors are estimated with un-


S C ⫽S⫺ ␦ I* ⫹
J

兺 ␦*
j⫽1
j 冊 ⫽T⫹ ␦ SM ⫹␧ SN (10)
certainties.
If (X i ,r i ), and S share no common error sources, then the un-
certainty U E in the comparison error can be expressed as

U S2 C N ⫽U I2C ⫹ 兺
j⫽1
J

U 2j C (11) U E2 ⫽ 冉 冊 冉 冊
⳵E
⳵D
2
2
UD ⫹
⳵E
⳵S
2
⫹U 2S ⫽U D
2
⫹U 2S (15)

or
Verification is based on equation 共10兲, which is put in the form
U E2 ⫽U D
2
⫹U SM
2
A ⫹U S PD ⫹U SN
2 2

冉 冊
(16)
J

S⫽S C ⫹ ␦ I* ⫹ 兺 ␦*
j⫽1
j (12)
Ideally, one would postulate that if the absolute value of E is
less than its uncertainty U E , then validation is achieved 共i.e., E is
‘‘zero’’ considering the resolution imposed by the ‘‘noise level’’
Equation 共12兲 expresses S as the corrected simulation value S C U E 兲. In reality, there is no known approach that gives an estimate
plus numerical errors. S C is also referred to as a numerical bench- of U SM A , so U E cannot be estimated. That leaves a more stringent

Journal of Fluids Engineering DECEMBER 2001, Vol. 123 Õ 795

Downloaded 12 Jan 2011 to 59.162.23.4. Redistribution subject to ASME license or copyright; see http://www.asme.org/terms/Terms_Use.cfm
validation test as the practical alternative. If the validation uncer- 3 Verification Procedures
tainty U V is defined as the combination of all uncertainties that we
In Section 2, the simulation numerical error and uncertainty
know how to estimate 共i.e., all but U SM A 兲, then
were decomposed into contributions from iteration number, grid
size, time step, and other parameters in Eqs. 共7兲 and 共8兲. In this
U V2 ⫽U E2 ⫺U SM
2
A ⫽U D ⫹U STE
2 2
(17) section, detailed verification procedures are given for estimation
of these contributions through convergence studies 共Section 3.1兲.
2
where U STE ⫽U S2 PD ⫹U SN
2
is the total estimated simulation uncer- Iterative 共Section 3.2兲 and parameter 共Sections 3.3–3.5兲 conver-
tainty, as shown in Fig. 1. gence studies are conducted using multiple solutions with system-
If 兩E兩 is less than the validation uncertainty U V , the combina- atic parameter refinement to estimate numerical errors and uncer-
tion of all the errors in D and S is smaller than the estimated tainties. Three convergence conditions are possible: 共i兲 monotonic
validation uncertainty and validation has been achieved at the U V convergence; 共ii兲 oscillatory convergence; and 共iii兲 divergence and
level. U V is the key metric in the validation process. U V is the are described in Sections 3.3, 3.4, and 3.5, respectively. For con-
validation ‘‘noise level’’ imposed by the uncertainties inherent in dition 共i兲, as already mentioned, errors and uncertainties are esti-
the data, the numerical solution, and the previous experimental mated using generalized RE. For condition 共ii兲, uncertainties are
data used in the simulation model. It can be argued that one can- estimated simply by attempting to bound error based on oscilla-
not discriminate once 兩E兩 is less than this; that is, as long as 兩E兩 is tion maximums and minimums. For condition 共iii兲, errors and un-
less than this, one cannot evaluate the effectiveness of proposed certainties cannot be estimated. As discussed below and later in
model ‘‘improvements.’’ On the other hand, if 兩 E 兩 ⰇU V one could Section 5, there are many issues in estimating errors and uncer-
argue that probably E⬇ ␦ SM A . tainties for practical applications.
Oberkamp and Trucano 关14兴 have criticized Coleman and Stern
关6兴 for fact that U V excludes U SM A . As already acknowledged, 3.1 Convergence Studies. Iterative and parameter conver-
there is no known way for directly estimating U SM A . However, gence studies are conducted using multiple 共m兲 solutions and sys-
the present approach does provide a more stringent validation tematic parameter refinement by varying the kth input parameter
metric U V which sets the level that validation can be achieved as ⌬x k while holding all other parameters constant. The present work
the root sum square of the experimental U D and the total esti- assumes input parameters can be expressed such that the finest
mated simulation U STE uncertainties. Additionally, under certain resolution corresponds to the limit of infinitely small parameter
conditions, the simulation modeling error ␦ SM A itself can be esti- values. Many common input parameters are of this form, e.g., grid
mated, as further discussed in Section 4. Consideration of Eq. 共17兲 spacing, time step, and artificial dissipation. Additionally, a uni-
shows that 共1兲 the more uncertain the data 共greater U D 兲 and/or 共2兲 form parameter refinement ratio r k ⫽⌬x k 2 /⌬x k 1 ⫽⌬x k 3 /⌬x k 2
the more inaccurate the code 共greater U STE 兲, the easier it is to ⫽⌬x k m /⌬x k m⫺1 between solutions is assumed for presentation
validate a code, since the greater the uncertainties in the data and purposes, but not required as discussed later.
code predictions, the greater the noise level U V . Both Roache 关3兴 Careful consideration should be given to selection of uniform
and Oberkamp and Trucano 关14兴 have criticized Coleman and parameter refinement ratio. The most appropriate values for indus-
Stern 关6兴 for this fact. However, if the value of U V is greater than trial CFD are not yet fully established. Small values 共i.e., very
that designated as necessary in a research/design/development close to one兲 are undesirable since solution changes will be small
program, the required level of validation could not be achieved and sensitivity to input parameter may be difficult to identify com-
without improvement in the quality of the data, the code, or both. pared to iterative errors. Large values alleviate this problem; how-
Also, if U SN and U S PD are not estimated, but 兩E兩 is less than U D , ever, they also may be undesirable since the finest step size may
then a type of validation can be argued to have been achieved, but be prohibitively small 共i.e., require many steps兲 if the coarsest step
clearly as shown by the present methodology, at an unknown size is designed for sufficient resolution such that similar physics
level. are resolved for all m solutions. Also, similarly as for small val-
If the ‘‘corrected’’ approach of Eqs. 共3兲–共6兲 is used, then the ues, solution changes for the finest step size may be difficult to
equations, equivalent to Eqs. 共14兲 and 共17兲 are identify compared to iterative errors since iterative convergence is
more difficult for small step size. Another issue is that for param-
E C ⫽D⫺S C ⫽ ␦ D ⫺ 共 ␦ SM A ⫹ ␦ S PD ⫹␧ SN 兲 (18) eter refinement ratio other than r k ⫽2, interpolation to a common
location is required to compute solution changes, which intro-
for the corrected comparison error and duces interpolation errors. Roache 关3兴 discusses methods for
evaluating interpolation errors. However, for industrial CFD, r k
U V2 C ⫽U E2 C ⫺U SM
2
A ⫽U D ⫽U D ⫹U S C TE
2 2 2
(19) ⫽2 may often be too large. A good alternative may be r k ⫽&, as
it provides fairly large parameter refinement ratio and at least
for the corrected validation uncertainty where U S2 C TE ⫽U S2 PD enables prolongation of the coarse-parameter solution as an initial
guess for the fine-parameter solution.
⫹U S2 C N is the total estimated corrected simulation uncertainty,
Equation 共12兲 is written for the kth parameter and mth solution
also shown in Fig. 1. Note that S C and E C can be either larger or as
smaller than their counterparts S and E, but U E C and U V C should
f


be smaller than U E and U V , respectively, since U S C N should be
S k m ⫽S C ⫹ ␦ I*k ⫹ ␦ *
km⫹ ␦ *j m (20)
smaller than U SN . m j⫽1,j⫽k
If there is a programmatic validation requirement, there is an-
other uncertainty U reqd that must be considered since validation is Iterative convergence must be assessed and S k m corrected for it-
required at that uncertainty level or below. Interpretation of the erative errors prior to evaluation of parameter convergence since
meaning of the relative magnitudes of E 共or E C 兲, U reqd and U V the level of iterative convergence may not be the same for all m
共or U V C 兲 and of the implications on the possibility of estimating solutions used in the parameter convergence studies. Equation
␦ SM A are discussed in Section 4. Additional discussion is provided 共20兲 shows that iterative errors ␦ I*k must be accurately estimated
m
in Coleman and Stern 关6兴 on: estimating U S PD ; estimating U D for or negligible in comparison to ␦ * k m for accurate convergence stud-
the data point (X i ,r i ), including both the experimental uncer-
tainty in r i and the additional uncertainties in r i arising from the ies and that they should be considered within the context of con-
experimental uncertainties in the measurements of the n indepen- vergence studies for each input parameter. Methods for estimating
dent variables (X j ) i in X i ; and for validation of a CFD code, U I or ␦ I* and U I C are described in Section 3.2.2.
multiple codes and/or models, and prediction of trends. With ␦ I*k evaluated, S k m is corrected for iterative errors as
m

796 Õ Vol. 123, DECEMBER 2001 Transactions of the ASME

Downloaded 12 Jan 2011 to 59.162.23.4. Redistribution subject to ASME license or copyright; see http://www.asme.org/terms/Terms_Use.cfm
J gence study on the amplitude distribution of the Fourier modes. In
Ŝ k m ⫽S k m ⫺ ␦ I*k ⫽S C ⫹ ␦ k*m ⫹
m

j⫽1,j⫽k
␦ *j m (21) principle, this approach would remove the problem of ill-
conditioning of the convergence ratio, R k .
Ŝ k m can be calculated for both integral 共e.g., resistance coeffi- 3.2 Iterative Convergence. The number of order magni-
cients兲 and point 共e.g., surface pressure, wall-shear stress, and tude drop and final level of solution residual 共or residual imbal-
velocity兲 variables. Ŝ k m can be presented as an absolute quantity ance兲 can be used to determine stopping criteria for iterative so-
共i.e., non-normalized兲 or normalized with the solution as a per- lution techniques. Iterative convergence to machine zero is
centage change; however, if the solution value is small, a more desirable, but for complex geometry and conditions it is often not
appropriate normalization may be the range of the solution. possible. Three or four orders of magnitude drop in solution re-
Convergence studies require a minimum of m⫽3 solutions to sidual to a level of 10⫺4 is more likely for these cases. Methods
evaluate convergence with respect to input parameter. Note that for estimation of iterative errors and uncertainties can be based on
m⫽2 is inadequate, as it only indicates sensitivity and not con- graphical, as discussed below, or theoretical approaches and are
vergence, and that m⬎3 may be required. Consider the situation dependent on the type of iterative convergence: 共a兲 oscillatory; 共b兲
convergent; or 共c兲 mixed oscillatory/convergent.
for 3 solutions corresponding to fine Ŝ k l , medium Ŝ k 2 , and coarse
For oscillatory iterative convergence 共a兲, the deviation of the
Ŝ k 3 values for the kth input parameter. Solution changes ␧ for variable from its mean value provides estimates of the iterative
medium-fine and coarse-medium solutions and their ratio R k are uncertainty based on the range of the maximum S U and minimum
defined by S L values

␧ k 21⫽Ŝ k 2 ⫺Ŝ k 1
U I⫽ 冏 1
共 S ⫺S L 兲
2 U
冏 (24)
␧ k 32⫽Ŝ k 3 ⫺Ŝ k 2 (22)
For convergent iterative convergence 共b兲, a curve-fit of an ex-
R k ⫽␧ k 21 /␧ k 32 ponential function can be used to estimate U I or ␦ I* and U I C as
the difference between the value and the exponential function
Three convergence conditions are possible: from a curve fit for large iteration number CF ⬁
共 i 兲 Monotonic convergence: 0⬍R k ⬍1 U I ⫽ 兩 S⫺CF ⬁ 兩

共 ii 兲 Oscillatory convergence: R k ⬍0 1 (23) ␦ I*k ⫽S⫺CF ⬁ ,U I C ⫽0 (25)


m

共 iii 兲 Divergence: R k ⬎1 For mixed convergent/oscillatory iterative convergence 共c兲, the


amplitude of the solution envelope decreases as the iteration num-
For monotonic convergence 共i兲, generalized RE is used to esti-
ber increases, the solution envelope is used to define the maxi-
mate U k or ␦ * k and U k C . Methods for estimating errors and un- mum S U and minimum S L values in the Ith iteration, and to esti-
certainties for condition 共i兲 are described in Section 3.3. mate U I or ␦ I* and U I C
For oscillatory convergence 共ii兲, the solutions exhibit oscilla-
tions, which may be erroneously identified as condition 共i兲 or 共iii兲.
This is apparent if one considers evaluating convergence condi-
tion from three points on a sinusoidal curve 共Coleman et al. 关15兴兲.
U I⫽ 冏 1
共 S ⫺S L 兲
2 U
冏 (26)
Depending on where the three points fall on the curve, the condi- 1
tion could be incorrectly diagnosed as either monotonic conver- ␦ I*k ⫽S⫺ 共 S U ⫺S L 兲 ,U I C ⫽0
m 2
gence or divergence. Methods discussed here for estimating un-
certainties U k for condition 共ii兲 require more than m⫽3 solutions An increase in the amplitude of the solution envelope as the
and are described in Section 3.4. iteration number increases indicates that the solution is divergent.
For divergence 共iii兲, the solutions diverge and errors and uncer- Estimates of the iterative error based on theoretical approaches
tainties cannot be estimated. Additional remarks are given in Sec- are presented in Ferziger and Peric 关17兴 and involve estimation of
tion 3.5. the principal eigenvalue of the iteration matrix. The approach is
Determination of the convergence ratio R k for point variables relatively straightforward when the eigenvalue is real and the so-
can be problematic since solution changes ␧ k 21 and ␧ k 32 can both lution is convergent. For cases in which the principal eigenvalue is
go to zero 共e.g., in regions where the solution contains an inflec- complex and the solution is oscillatory or mixed, the estimation is
tion point兲. In this case, the ratio becomes ill conditioned. How- not as straightforward and additional assumptions are required.
ever, the convergence ratio can be used in regions where the so-
lution changes are both non-zero 共e.g., local solution maximums 3.3 Monotonic Convergence: Generalized Richardson Ex-
or minimums兲. Another approach is to use a global convergence trapolation. For monotonic convergence, i.e., condition 共i兲 in
ratio R k , which overcomes ill conditioning, based on the L2 norm Eq. 共23兲, generalized RE is used to estimate U k or ␦ *k and U k C .
of the solution changes, i.e., 具 R k 典 ⫽ 储 ␧ k 21储 2 / 储 ␧ k 32储 2 . 具 典 is used to RE is generalized for J input parameters and concept of correction
denote an averaged value and 储 ␧ 储 2 ⫽ 关 兺 i⫽1 N
␧ i2 兴 1/2 denotes the L2 factors based on analytical benchmarks is introduced. More de-
norm of solution change over the N points in the region of inter- tailed derivations are provided by Stern et al. 关13兴.
est. Caution should be exercised when defining the convergence As already mentioned, since Stern et al. 关13兴 there has been
ratio from the ratio of the L2 norm of solution changes because nearly two years experience with present approach, especially
the oscillatory condition (R k ⬍1) cannot be diagnosed since 具 R k 典 through ITTC community and Gothenburg 2000 Workshop on
will always be greater than zero. Local values of R k at solution CFD in Ship Hydrodynamics. In particular, detailed verification
maximums or minimums should also be examined to confirm the procedures have been the focus of attention 共Eca and Hoekstra
convergence condition based on an L2 norm definition. An alter- 关18兴; Ebert and Gorski 关19兴兲. After some background for general-
nate approach suggested by Hoekstra et al. 关16兴 is to transform the ized RE is given, two approaches for estimating errors and uncer-
spatial profile to wave number space and to perform a conver- tainties are presented and are based on 共i兲 correction factors pro-
posed in the current paper and 共ii兲 factor of safety approach
1
As discussed in the text that follows, 0⬍R k ⬍1 and R k ⬎1 may also occur for proposed by Roache 共1998兲. Finally, a discussion of fundamental
the oscillatory condition. and practical issues for verification is provided.

Journal of Fluids Engineering DECEMBER 2001, Vol. 123 Õ 797

Downloaded 12 Jan 2011 to 59.162.23.4. Redistribution subject to ASME license or copyright; see http://www.asme.org/terms/Terms_Use.cfm
Background for Generalized RE. Generalized RE begins with medium input parameters r k 32 can be used to estimate ␦ * k 1 from
Eq. 共21兲. The error terms on the right-hand side of Eq. 共21兲 are of Eq. 共29兲, provided that Eq. 共30兲 for estimating order of accuracy is
known form 共i.e., power series expansion with integer powers of modified as
⌬x k 兲 based on analysis of the modified and numerical error equa-
tions which is written below as a finite sum 共i.e., error estimate兲 ln共 ␧ k 32 /␧ k 21兲 1 p p
and for the kth parameter and mth solution p k⫽ ⫹ 关 ln共 r k k ⫺1 兲 ⫺ln共 r k k ⫺1 兲兴
ln共 r k 21兲 ln共 r k 21兲 32 21
n

兺 共 ⌬x
i兲
p共
␦ *k m ⫽ km兲 k g 共ki 兲 (27) (31)
i⫽1
For situations when r k 21⫽r k 32, Eq. 共31兲 is a transcendental equa-
n⫽number of terms retained in the power series, powers p (ki ) cor-
tion implicitly defining p k and must be solved iteratively. If r k 21
respond to order of accuracy 共for the ith term兲, and g (ki ) are re-
⫽r k 32, Eq. 共31兲 degenerates to Eq. 共30兲.
ferred to as ‘‘grid’’ functions which are a function of various or-
ders and combinations of derivatives of S with respect to x k . It is Estimating Errors and Uncertainties Using Generalized RE
assumed that the power series in Eq. 共27兲 is convergent 共i.e., the With Correction Factors. Results from the numerical solution of
finite sum convergence to the infinite series value as more terms the one-dimensional 共1D兲 wave and two-dimensional 共2D兲
are included兲. Substituting Eq. 共27兲 into Eq. 共21兲 results in Laplace equation analytical benchmarks show that Eq. 共29兲 has
n J the correct form, but the order of accuracy is poorly estimated by
兺 兺
共i兲
Ŝ k m ⫽S C ⫹ 共 ⌬x k m 兲 p k g 共ki 兲 ⫹ ␦ *j m (28) Eq. 共30兲 except in the asymptotic range. Analysis of the results
i⫽1 j⫽1,j⫽k suggests the concept of correction factors, which provide a quan-
titative metric to determine proximity of the solutions to the
Subtraction of multiple solutions where input parameter ⌬x k is asymptotic range, account for the effects of higher-order terms,
uniformly refined eliminates the ␦ *j m terms in Eq. 共28兲 since ␦ *j m is and are used for defining and estimating errors and uncertainties.
independent of ⌬x k and provides equations for S C , p (ki ) , and g (ki ) . Details are provided in Appendix A.
This assumes p (ki ) and g (ki ) are also independent of ⌬x k . Since Multiplication of Eq. 共29兲 by a correction factor C k provides an
each term 共i兲 contains 2 unknowns, m⫽2n⫹1 solutions are re- estimate for ␦ *
k 1 accounting for the effects of higher-order terms

冉 冊
quired to estimate the numerical benchmark S C and the first n
terms in the expansion in Eq. 共28兲 共i.e., for n⫽1, m⫽3 and for ␧ k 21
n⫽2, m⫽5, etc兲. The accuracy of the estimates depends on how ␦ *k 1 ⫽C k ␦ RE
* ⫽C k p (32)
k 1 r k k ⫺1
many terms are retained in Eq. 共27兲, the magnitude 共importance兲
of the higher-order terms, and the validity of the assumption that If solutions are in the asymptotic range, correction of Eq. 共29兲 is
p (ki ) and g (ki ) are independent of ⌬x k . For sufficiently small ⌬x k , not required 关i.e., C k ⫽1 so that Eqs. 共29兲 and 共32兲 are equivalent兴.
the solutions are in the asymptotic range such that higher-order For solutions outside the asymptotic range, C k ⬍1 or C k ⬎1 indi-
terms are negligible and the assumption that p (ki ) and g (ki ) are in- cates that the leading-order term over predicts 共higher-order terms
dependent of ⌬x k is valid. However, achieving the asymptotic net negative兲 or under predicts 共higher-order terms net positive兲
range for practical geometry and conditions is usually not possible the error, respectively. The estimate given by Eq. 共32兲 includes
and m⬎3 is undesirable from a resources point of view; therefore, both sign and magnitude and is used to estimate U k or ␦ * k and U k C
methods are needed to account for effects of higher-order terms depending on how close the solutions are to the asymptotic range
for practical application of RE. Additionally, methods may be 共i.e., how close C k is to 1兲 and one’s confidence in Eq. 共32兲. There
needed to account for possible dependence of p (ki ) and g (ki ) on are many reasons for lack of confidence, especially for complex
⌬x k , although not addressed herein. Usually ␦ k* is estimated for three-dimensional flows.
the finest value of the input parameter, i.e., ␦ * k ⫽ ␦ k*1 corresponding
For C k sufficiently less than or greater than 1 and lacking con-
to the finest solution S k 1 . fidence, U k is estimated, but not ␦ * k and U k c . Equation 共32兲 is

With three solutions (m⫽3), only the leading-order term of used to estimate the uncertainty by bounding the error ␦ * k by the
Eq. 共27兲 can be estimated. Solution of the three equations for S C , sum of the absolute value of the corrected estimate from RE and
p (ki ) , and g (ki ) yields estimates for the error ␦ * the absolute value of the amount of the correction
k 1 and order-of-
accuracy p k U k ⫽ 兩 C k ␦ RE
* 兩 ⫹ 兩 共 1⫺C k 兲 ␦ RE
k
* 兩
k
(33)
1 1
␧ k 21
␦ *k 1 ⫽ ␦ RE
* ⫽ p (29) For C k sufficiently close to 1 and having confidence, ␦ * k and
k r k k ⫺1 U k c are estimated. Equation 共32兲 is used to estimate the error ␦ *
k ,
1

ln共 ␧ k 32 /␧ k 21兲 which can then also be used in the calculation of S c 关in Eq. 共10兲兴.
p k⫽ (30) The uncertainty in the error estimate is based on the amount of the
ln共 r k 兲 correction
Solving for the first-order term is relatively easy since evaluation U k C ⫽ 兩 共 1⫺C k 兲 ␦ RE
* 兩 (34)
of Eqs. 共29兲 and 共30兲 only requires that the m⫽3 solutions are k 1

monotonically convergent, even if the solutions are far from the


Note that in the limit of the asymptotic range, C k ⫽1, ␦ *
k ⫽␦*
asymptotic range and Eqs. 共29兲 and 共30兲 are inaccurate. With so- k1

lutions from five systematically refined input parameters (m ⫽ ␦ RE k , and U k C ⫽0.


*
⫽5), more complicated expressions can be derived to estimate
1
Two definitions for the correction factor were developed. The
the first two terms of the power series expansion. However, their first is based on solution of Eq. 共32兲 for C k with ␦ RE
* based on
range of applicability is more restrictive since all five solutions k 1

must be both monotonically convergent and sufficiently close to Eq. 共29兲 but replacing p k with the improved estimate p k est
the asymptotic range for the expressions to be used. p
As previously mentioned, solutions from three values of input r k k ⫺1
parameter where the refinement ratio between the medium and C k⫽ p (35)
fine input parameters r k 21 is not equal to that between coarse and r k k est ⫺1

798 Õ Vol. 123, DECEMBER 2001 Transactions of the ASME

Downloaded 12 Jan 2011 to 59.162.23.4. Redistribution subject to ASME license or copyright; see http://www.asme.org/terms/Terms_Use.cfm
Similarly, the second is based on a two-term estimate of the power been used to address some of these fundamental issues while oth-
series which is used to estimate ␦ RE* where p k and q k are re-
k
ers need further research. Although both correction factor and
1
factor of safety approaches were presented, the authors advocate
placed with p k est and q k est the use of former. Results from the numerical solution of analytic
q p p p benchmarks show that the factor of safety approach is overly con-
共 ␧ k 23 /␧ k 12⫺r k k est 兲共 r k k ⫺1 兲 共 ␧ k 23 /␧ k 12⫺r k k est 兲共 r k k ⫺1 兲 servative, especially when the solutions approach the asymptotic
C k⫽ p q p ⫹ p q q range 共Appendix A兲. This is in contrast to the variable correction
共 r k k est ⫺r k k est 兲共 r k k est ⫺1 兲 共 r k k est ⫺r k k est 兲共 r k k est ⫺1 兲
factor approach proposed in Eqs. 共33兲 and 共34兲, where the uncer-
(36) tainty in the error estimate correctly goes to zero as the asymptotic
p k est and q k est are estimates for limiting orders of accuracy of the range is approached because C k →1. Admittedly, others have rec-
ommended the factor of safety approach, e.g., Eca and Hoekstra
first and second terms of the error expansion equation 共27兲 as
关18兴, although examination of their results as with our own analy-
spacing size goes to zero and the asymptotic range is reached.
sis indicates that such estimates are overly conservative.
Equation 共35兲 roughly accounts for the effects of higher-order
For practical applications, especially complex flows with rela-
terms by replacing p k with p k est thereby providing an improved
tively coarse grids, solutions may be far from asymptotic range
single-term estimate. Equation 共36兲 more rigorously accounts for such that some variables are convergent while others are oscilla-
higher-order terms since it is derived from the two-term estimate tory or even divergent. Order of accuracy and therefore correction
with first and second term order of accuracy p (k1 ) and p (k2 ) replaced factors and factors of safety may display large variability indicat-
by p k est and q k est . Equation 共36兲 simplifies to Eq. 共35兲 in the limit ing the need for finer grids. Clearly, more than 3 grids are required
of the asymptotic range. Both correction factors only require so- to estimate errors and uncertainties for such cases. Eca and Hoek-
lutions for three parameter values. The estimated values p k est and stra 关18兴 suggest a least-squares approach to estimate the error by
q k est can be based either on the assumed theoretical order of ac- computing the three unknown parameters from RE when more
than three solutions are available. The behavior of the asymptotic
curacy p k th and q k th or solutions for simplified geometry and con- range was successfully demonstrated for simpler analytical bench-
ditions. In either case, preferably including the effects of grid marks in Appendix A. However, the existence and behavior of the
stretching. asymptotic range for practical problems has not been demon-
In Appendix A, exact 共A兲 and numerical 共S兲 solutions are used strated due to lack of sufficiently refined grids, number of solu-
to compare the true simulation error (A⫺S) to 共i兲 an uncorrected tions to assess variability, and available resources, among other
three-grid error estimate using Eq. 共29兲 and 共ii兲 corrected esti- issues. Another practical issue involves selecting and maintaining
mates based on Eq. 共32兲 with correction factor defined by Eq. 共35兲 appropriate parameter refinement ratio and resources for obtaining
or 共36兲. Correction of error estimates with both definitions of C k solutions with sufficient parameter refinement as well as number
results in improved error estimates. Also, uncertainty estimates of solutions. Lastly, interpretation of results is an issue since, as
using Eq. 共33兲 with correction factor defined by Eq. 共35兲 or 共36兲 already mentioned, there is limited experience and no known
are shown to bound the true simulation error (A⫺S), while un- solutions for practical applications in the asymptotic range for
certainty estimates using Eq. 共34兲 are shown to bound the differ- guidance.
ence between the corrected solution and the truth (S c -T). Addi- The present verification procedures represent the most rational
tional testing of expressions for C k given by Eqs. 共35兲 and 共36兲 is approach presently known. However, alternative strategies for in-
needed and development of improved expressions within the pro- cluding effects of higher-order terms may be just as viable, e.g.,
posed general framework is certainly possible. treatment of the power series exponents as known integers as
Estimating Uncertainties Using Generalized RE With Factors proposed by Oberkampf and investigated by Eca and Hoekstra
of Safety. In Roache 关3兴, a GCI approach is proposed where a 关18兴. Once available, improved verification procedures can be eas-
standard three-grid error estimate from RE is multiplied by a fac- ily incorporated into the present overall verification and validation
tor of safety F S to bound the simulation error methodology. These issues are discussed further in Section 5 Con-
clusions and Recommendations and in Part 2 共Wilson et al. 关12兴兲.
U k ⫽F S 兩 ␦ RE
* 兩
k
(37) 3.4 Oscillatory Convergence. For oscillatory convergence,
1
i.e., condition 共ii兲 in Eq. 共23兲, uncertainties can be estimated, but
Note that Eq. 共37兲 with factor of safety differs significantly from not the signs and magnitudes of the errors. Uncertainties are esti-
Eq. 共34兲. Herein C k ⫽C k (␧,r k ,p k ,p k est ,q k est ), in contrast to Eq. mated based on determination of the upper (S U ) and lower (S L )
共37兲 where C k is a constant referred to as a factor of safety F S . bounds of solution oscillation, which requires more than m⫽3
The exact value for factor of safety is somewhat ambiguous and solutions. The estimate of uncertainty is based on half the solution
Roache 关3兴 recommends 1.25 for careful grid studies and 3 for range
cases in which only two grids are used.
Although not proposed in Roache 关3兴, the factor of safety ap- 1
U k ⫽ 共 S U ⫺S L 兲 (39)
proach can be used for situations where the solution is corrected 2
with an error estimate from RE. Equation 共29兲 is used to estimate
␦ k* and the uncertainty in that error estimate is given by 3.5 Divergence. For divergence, i.e., condition 共iii兲 in Eq.
共23兲, errors and or uncertainties can not be estimated. The prepa-
U k C ⫽ 共 F S ⫺1 兲 兩 ␦ RE
* 兩 (38) ration and verification steps must be reconsidered. Improvements
k 1 in iterative convergence, parameter specification 共e.g., grid qual-
With this approach, a fixed percentage of a three-grid error ity兲, and/or CFD code may be required to achieve converging or
oscillatory conditions.
estimate 共e.g., 25% ␦ RE
* for F S ⫽1.25兲 is used to define the un-
k 1
certainty of the error estimate regardless of how close solutions 4 Validation Procedures
are to the asymptotic range.
In Section 2, an approach for assessing the simulation modeling
Discussion of Fundamental and Practical Issues. Fundamen- uncertainty was presented where for successful validation, the
tal and practical issues for verification are discussed in this sec- comparison error, E is less than the validation uncertainty, U V
tion. Fundamental issues include convergence of power series given by Eqs. 共17兲 and 共19兲 for uncorrected and corrected solu-
equation 共27兲, assumptions that p (ki ) and g (ki ) are independent of tions, respectively. In this section, validation procedures are pre-
⌬x k , and estimating p k est . Solution of analytical benchmarks has sented through discussions in Section 4.1 on interpretation of

Journal of Fluids Engineering DECEMBER 2001, Vol. 123 Õ 799

Downloaded 12 Jan 2011 to 59.162.23.4. Redistribution subject to ASME license or copyright; see http://www.asme.org/terms/Terms_Use.cfm
validation results and in Section 4.2 on use of corrected simula- 5 Conclusions
tion results. As previously mentioned, Coleman and Stern 关6兴 pro-
The present comprehensive approach to verification and valida-
vide additional discussion on validation procedures.
tion methodology and procedures sets forth concepts, definitions,
and equations derived for simulation errors and uncertainties,
4.1 Interpretation of the Results of a Validation Effort. which provide a well-founded mathematical framework. The ap-
First, consider the approach in which the simulation numerical proach should have applicability to a fairly broad range of CFD
error is taken to be stochastic and thus the uncertainty U SN is codes, including RANS, Navier-Stokes, Euler, boundary-element
estimated. From a general perspective, if we consider the three methods, and others. However, clearly much more work is needed
variables U V , 兩E兩, and U reqd there are six combinations 共assuming for other CFD codes 共such as large-eddy simulations兲, additional
none of the three variables are equal兲: error sources, and alternative error and uncertainty estimation
methods, e.g., single-grid methods and both results for additional
1. 兩 E 兩 ⬍U V ⬍U reqd analytical benchmarks 共especially for nonlinear equations and us-
ing stretched grids兲 for improved definitions of correction factors
2. 兩 E 兩 ⬍U reqd ⬍U V and estimates of orders of accuracy, and alternative strategies to
account for the effects of higher-order terms in RE. Improved
3. U reqd ⬍ 兩 E 兩 ⬍U V
verification procedures once available can be easily incorporated
4. U V ⬍ 兩 E 兩 ⬍U reqd into the present overall verification and validation methodology.
Furthermore, more experience is needed through application for
5. U V ⬍U reqd ⬍ 兩 E 兩 different codes and geometry and conditions, especially for prac-
tical applications.
6. U reqd ⬍U V ⬍ 兩 E 兩 (40) As mentioned in the Introduction, present verification and vali-
dation methodology and procedures were recommended and used
In cases 1, 2, and 3, 兩 E 兩 ⬍U V ; validation is achieved at the U V
at the recent Gothenburg 2000 Workshop on CFD in Ship Hydro-
level; and the comparison error is below the noise level, so at-
dynamics 共Larsson et al. 关11兴兲. 22 participating research groups
tempting to estimate ␦ SM A is not feasible from an uncertainty
from 12 countries and 19 different RANS codes were used for
standpoint. In case 1, validation has been achieved at a level
simulations of 3 test cases representing tanker, container, and sur-
below U reqd , so validation is successful from a programmatic
face combatant hull forms. Most groups implemented the recom-
standpoint.
mended procedures, but lack of familiarity with the procedures
In cases 4, 5, and 6, U V ⬍ 兩 E 兩 , so the comparison error is above
and use of coarse grids led to difficulties. Coarser grid solutions
the noise level and using the sign and magnitude of E to estimate
are far from the asymptotic range and show variability such that
␦ SM A is feasible from an uncertainty standpoint. If U V Ⰶ 兩 E 兩 , then
not all variables display monotonic convergence and oscillatory
E corresponds to ␦ SM A and the error from the modeling assump-
convergence and even divergence is evident. For monotonic con-
tions can be determined unambiguously. In case 4, validation is
vergence, variability in the estimated order of accuracy was ob-
successful at the 兩E兩 level from a programmatic standpoint.
served for some cases. The current 1 million point grids are
Now consider the approach in which the simulation numerical
clearly insufficient for more complex hull forms such as the tanker
error is taken to be deterministic and thus ␦ SN * and the uncertainty
and an order of magnitude increase in points may be required to
U V C are estimated. A similar set of comparisons as those in Eq. remove variability and achieve monotonic convergence for most
共40兲 can be constructed using 兩 E C 兩 , U V C , and U reqd . Since E C variables. In spite of difficulties, the effort was beneficial in en-
can be larger or smaller than E, but U V C should always be less abling quantitative evaluation of levels of verification and valida-
than U V , the results for a given corrected case are not necessarily tion, increasing familiarity with verification and validation proce-
analogous to those for the corresponding uncorrected case. That dures, interpretation of results, and identification of grid
is, a variable can be validated in the corrected but not in the requirements for decreasing levels of errors and uncertainties.
uncorrected case, or vice versa. For cases 4, 5, and 6 in which Careful examination of verification results even for relatively
U V C ⬍ 兩 E C 兩 , one can argue that E C is a better indicator of ␦ SM A coarse grid solutions provides a road map towards achieving ac-
ceptable levels of verification.
than is E, assuming that one’s confidence in using the estimate
Verification and validation methodology and procedures should
␦ SN
* is not misplaced.
be helpful in guiding future developments in CFD through docu-
mentation, verification, and validation studies and in transition of
4.2. Use of Corrected Versus Uncorrected Simulation Re- CFD codes to design through establishment of credibility. Pre-
sults. As previously stated in Section 3.3, the requirements for sumably, with a sufficient number of documented, verified, and
correcting the solution are that the correction factor be close to validated solutions along with selected verification studies, a CFD
one and that confidence in solutions exist. Since the variability of code can be accredited for a certain range of applications. The
the order of accuracy cannot be determined from solutions on contribution of the present work is in providing methodology and
three grids, confidence is difficult to establish in this case. As a procedures for the former, which hopefully will help lead to the
result, caution should be exercised when correcting solutions us- latter.
ing information from only three grids. Part 2 provides an example for RANS simulations for a cargo/
If a validation using the corrected approach is successful at a container ship where issues with regard to practical application of
set condition, then if one chooses to associate that validation un- the methodology and procedures and interpretation of verification
certainty level with the simulation’s prediction at a neighboring and validation results are discussed 共Wilson et al. 关12兴兲.
condition that prediction must also be corrected. That means
enough runs are required at the new condition to allow estimation
of the numerical errors and uncertainties. If this is not done, then Acknowledgments
the comparison error E and validation uncertainty U V correspond- This research was sponsored by the Office of Naval Research
ing to the use of the uncorrected S and its associated 共larger兲 U SN under Grants N00014-96-1-0018, N00014-97-1-0014, and
should be the ones considered in the validation with which one N00014-97-1-0151 under the administration of Dr. E. P. Rood.
wants to associate the prediction at a new condition. 共Whether to The authors gratefully acknowledge Dr. Rood and other col-
and how to associate an uncertainty level at a validated condition leagues, especially Prof. W. G. Steele and Dr. H. Raven, who
with a prediction at a neighboring condition is very much unre- made significant contributions through insightful discussions and
solved and is justifiably the subject of much debate at this time.兲 comments on early drafts. The recent Masters and Ph.D. theses of

800 Õ Vol. 123, DECEMBER 2001 Transactions of the ASME

Downloaded 12 Jan 2011 to 59.162.23.4. Redistribution subject to ASME license or copyright; see http://www.asme.org/terms/Terms_Use.cfm
Dr. B. Chen, Mr. G. Dolphin and Dr. S. H. Rhee, all at The
University of Iowa, Department of Mechanical Engineering, were
helpful both in the development and testing of the present verifi-
cation and validation methodology and procedures.

Nomenclature
Ck ⫽ correction factor
D ⫽ benchmark data
E,E C ⫽ comparison error, corrected
pk ⫽ order of accuracy
Rk ⫽ parameter refinement ratio
S,S C ⫽ simulation result, corrected
T ⫽ truth
U ⫽ uncertainty estimate
UD ⫽ data uncertainty
U E ,U E C ⫽ comparison error uncertainty, corrected
UI ⫽ iteration uncertainty
Fig. 2 Verification results for first-order numerical solution of
U P ,U P C ⫽ parameter uncertainty 共e.g., grid size G and 1D wave equation. „a… Comparison of true error A À S to esti-
time step T兲, corrected mates from RE, „b… correction factor, and „c… comparison of
U reqd ⫽ programmatic validation requirement 円 A À S C 円 and U SCN , and „d… comparison of 円 A À S 円 and U SN .
U S ,U S C ⫽ simulation uncertainty, corrected
U SM ⫽ simulation modeling uncertainty
U SM A ⫽ simulation modeling assumption uncertainty
U S PD ⫽ simulation uncertainty due to use of previous
data tion; and 共ii兲 a second-order implicit method with second-order
U STE ,U S C TE ⫽ simulation total estimated numerical uncertainty central spatial discretization. Since trends from both schemes are
U SN ,U S C N ⫽ simulation numerical uncertainty, corrected similar, only the results from the first-order scheme are presented.
A combined grid size and time step study was performed where
U V ,U V C ⫽ validation uncertainty, corrected ten solutions were obtained by successively doubling both the grid
⌬x k ⫽ increment in kth input parameter 共e.g., grid size and time step such that ⌬t/⌬x⫽0.5 for all solutions. With this
G and time step T兲 approach, solutions changes are used to estimate total 共temporal

⫽ error
and spatial兲 simulation errors and uncertainties. Accordingly, the
␦* ⫽ error estimate with sign and magnitude generic subscript ‘k’ appearing in expressions for errors and un-
␦ I , ␦ I*
⫽ iteration error, estimate certainties in Section 3.3 is replaced with ‘SN’ in this section
␦ P , ␦ *P⫽ parameter error, estimate where appropriate.
␦ S , ␦ SC⫽ simulation error, corrected
Errors, Uncertainties, and Correction Factors. The con-
␦ SN ⫽ simulation numerical error
␦ SM A ⫽ simulation modeling assumption error cept of a multiplication correction factor was introduced in Sec-
␧ ⫽ solution change tion 3.3. The correction factor C k was used to define the numerical
␧ SN ⫽ uncertainty in Eq. 共33兲 or when conditions permit to improve error
error in ␦ *
estimates in Eq. 共32兲 and to define the uncertainty in that error
estimate in Eq. 共34兲. Error and uncertainty estimates given by Eqs.
Appendix A. Analytical Benchmarks 共32兲–共34兲 are tested by numerical solution of analytical bench-
The use of analytical benchmarks for development of the con- marks as well as development of expressions for correction factor.
cept of correction factors as discussed in Section 3.3 is presented Figure 2共a兲 compares the true simulation error E to the three-
in this Appendix. For analytical benchmarks, the modeling error is grid error estimate ␦ RE
* from Eq. 共29兲 versus step size at one
1
zero such that the simulation error is solely due to numerical error. spatial location 共x⫽1 since maximums of numerical error occur
Results are obtained for two analytical benchmarks one- there兲. The three-grid estimate accurately estimates the true error
dimensional 共1D兲 wave and two-dimensional 共2D兲 Laplace equa- E for smaller step sizes, but over predicts E for larger step sizes.
tions. The results for the 2D Laplace equation were qualitatively Closer examination reveals that Eq. 共29兲 over estimates the error
similar to those for the 1D wave equation, which are presented. because Eq. 共30兲 under estimates the order of accuracy, as also
Exact solutions from analytical benchmarks are used to determine shown in Fig. 2共a兲.
the exact simulation numerical error which is compared to esti-
Two definitions for C k were investigated. The first is based on
mates from RE, including use of correction factors. More details
are provided in Stern et al. 关13兴, including single grid error solving equation 共32兲 for C k with ␦ RE* defined in Eq. 共29兲 but
k 1
estimates. replacing p k with the improved estimate p k est , which is provided
Verification of Analytical Benchmarks. For verification us- by Eq. 共35兲 where p k est is an estimate of the limiting order of
ing an analytical benchmark, the simulation error and uncertainty accuracy of the first term of the error expansion equation 共27兲.
are given by ␦ S ⫽S⫺A⫽ ␦ SN and U 2S ⫽U SN 2
, while the corrected Similarly, the second definition of correction factor is based on
simulation error and uncertainty are given by ␦ S C ⫽S C ⫺A⫽␧ SN estimating ␦ RE
* using the first two terms of the powers series and
k1
and U S2 C ⫽U S2 C N . Simulations are verified if 兩 E 兩 ⫽ 兩 A⫺S 兩 ⬍U SN replacing p k and q k with improved estimates p k est and q k est , which
and corrected simulations are verified if 兩 E C 兩 ⫽ 兩 A⫺S C 兩 ⬍U S C N . is provided by Eq. 共36兲 where p k est and q k est are estimates for
The first-order, linear 1D wave equation models the behavior of limiting orders of accuracy of the first and second terms of the
a more complicated 共nonlinear兲 partial differential equation. The error expansion equation 共27兲 as spacing size goes to zero and the
initial condition is prescribed by a Gaussian function centered at asymptotic range is reached. With this definition, correction fac-
x⫽0.0. Two discretization techniques were studied: 共i兲 first-order tors approach one in the limit of zero spacing size. The estimated
共Euler兲 explicit method with first-order upwind spatial discretiza- values p k est and q k est can be based either on the assumed theoret-

Journal of Fluids Engineering DECEMBER 2001, Vol. 123 Õ 801

Downloaded 12 Jan 2011 to 59.162.23.4. Redistribution subject to ASME license or copyright; see http://www.asme.org/terms/Terms_Use.cfm
ical order of accuracy p k th and q k th or solutions for simplified Laplace equation analytical benchmark. Their results are consis-
geometry and conditions. In either case, preferably including the tent with our own in showing that uncertainty estimates using Eq.
effects of grid stretching. 共33兲 always bounded the true error. Unlike our own results, their
Figure 2共a兲 also compares the true error E to 共i兲 an uncorrected results indicate that the uncertainty estimate from Eq. 共34兲 failed
three-grid error estimate using Eq. 共29兲 and 共ii兲 corrected esti- to bound the difference in the truth and numerical benchmark for
mates based on Eq. 共32兲 with correction factor defined by Eq. 共35兲 some grid triplets when the apparent order of accuracy was esti-
or 共36兲. Both estimates are closer to E than the uncorrected three mated to be larger than the theoretical value.
grid estimate ␦ RE
* , but for coarser grids C (k1 ) is somewhat too
1
small and C (k2 ) is slightly too large. Figure 2共b兲 shows the same References
trends, but directly compares the exact correction factor E/ ␦ RE * to 关1兴 Freitas, C. J., 1993, ‘‘Editorial Policy Statement on the Control of Numerical
Eqs. 共35兲 and 共36兲. In this case, C k ⬍1 indicates that the leading- Accuracy,’’ ASME J. Fluids Eng., 115, pp. 339–340.
关2兴 AIAA, 1998, Guide for the Verification and Validation of Computational Fluid
order term over predicts 共higher-order terms net negative兲 the er- Dynamics Simulations, G-077-1998.
ror. However, for the general case, C k is equally likely to be ⬍1 or 关3兴 Roache, P. J., 1998, Verification and Validation in Computational Science and
⬎1 depending whether the order of accuracy is approached from Engineering, Hermosa Publishers, Albuquerque, New Mexico.
below or above, respectively. C k ⬎1 indicates that the leading- 关4兴 Mehta, U. B., 1998, ‘‘Credible Computational Fluids Dynamics Simulations,’’
AIAA J., 36, pp. 665– 667.
order term under predicts 共higher-order terms net positive兲 the 关5兴 Stern, F., Paterson, E. G., and Tahara, Y., 1996, ‘‘CFDSHIP-IOWA: Computa-
error. Thus, for the general case the correction to the leading-term tional Fluid Dynamics Method for Surface-Ship Boundary Layers and Wakes
error estimate is equally likely to be positive or negative and can and Wave Fields,’’ Iowa Institute of Hydraulic Research, The University of
be used to define the simulation numerical uncertainty. Iowa, IIHR Report No. 381.
关6兴 Coleman, H. W., and Stern, F., 1997, ‘‘Uncertainties in CFD Code Validation,’’
For C k sufficiently close to 1 and having confidence, ␦ * k and ASME J. Fluids Eng., 119, pp. 795– 803. 共Also see ‘‘Authors’ Closure,’’
U k C are estimated. Correction factors C (k1 ) and C (k2 ) are used to ASME J. Fluids Eng., Vol. 120, Sept. 1998, pp. 635– 636.兲
关7兴 Coleman, H. W., and Steele, W. G., 1999, Experimentation and Uncertainty
estimate the error ␦ * k in Eq. 共32兲 which can then also be used in Analysis for Engineers, 2nd Edition, Wiley, New York, NY.
the calculation of S C 关in Eq. 共10兲兴 and uncertainty U k C in Eq. 共34兲. 关8兴 Rood, E. P., 1996, ‘‘Validation Strategy for RANS Computational Ship Hydro-
dynamics,’’ 2nd International Conference on Hydrodynamics, Hong Kong.
Figure 2共c兲 shows a comparison of 兩 A⫺S C 兩 and three uncertainty 关9兴 ITTC, 1996, 21st ITTC Proceedings, ‘‘Report of the Resistance Committee,’’
estimates: 共i兲 U S( 1C)N defined using C (k1 ) ; 共ii兲 U S( 2C)N defined using Bergen/Trondheim, Norway, September.
关10兴 ITTC, 1999, 22nd ITTC Proceedings, ‘‘Report of the Resistance Committee,’’
C (k2 ) ; and 共iii兲 U S C N from a factor of safety approach given by Eq. Seoul, Korea/Shanghai, China, Sept.
共38兲 with F S ⫽1.25. The results show that the uncertainty estimate 关11兴 Larsson, L., Stern, F., Bertram, V., 2000, ‘‘Gothenburg 2000 A Workshop on
Numerical Ship Hydrodynamics,’’ Chalmers University of Technology, Goth-
U S( 1C)N successfully bounds 兩 A⫺S C 兩 over the entire range of step enburg, Sweden, Sept.
关12兴 Wilson, R. V., Stern, F., Coleman, H., and Paterson, E., 2001, ‘‘Comprehensive
sizes and that U S( 2C)N is not conservative enough since U S( 2C)N ⬍ 兩 A Approach to Verification and Validation of CFD Simulations—Part 2: Appli-
⫺S C 兩 . The uncertainty estimate based on the factor of safety ap- cations for RANS Simulation of A Cargo Container Ship,’’ ASME J. Fluids
proach is not conservative enough for the coarsest two grids and is Eng., 123, published in the issue, pp. 803– 810.
关13兴 Stern, F., Wilson, R. V., Coleman, H., and Paterson, E., 1999, ‘‘Verification and
overly conservative by an order of magnitude for the four finest Validation of CFD Simulations,’’ Iowa Institute of Hydraulic Research, The
grids 共i.e., when solutions are in the asymptotic range兲. For C k University of Iowa, IIHR Report No. 407.
sufficiently less than or greater than 1 and lacking confidence, U k 关14兴 Oberkampf, W. L. and Trucano, T. G., 2000, ‘‘Validation Methodology in
is estimated, but not ␦ * (1)
k and U k C . Correction factors C k and C k
(2) Computational Fluid Dynamics,’’ AIAA Fluids 2000, Paper No. 2549, Denver,
CO.
are used to estimate the uncertainty in Eq. 共33兲 which is compared 关15兴 Coleman, H. W., Stern, F., Mascio, A. Di, and Campana, E., 2001, ‘‘The
to factor of safety approach given by Eq. 共37兲. Figure 2共d兲 shows Problem with Oscillatory Behavior in Grid Convergence Studies,’’ ASME J.
Fluids Eng., 123, No. 2, pp. 438 – 439.
that all three uncertainty estimates successfully bound the true 关16兴 Hoekstra, M., Eca, L., Windt, J., and Raven, H., 2000 ‘‘Viscous Flow Calcu-
error 兩 A⫺S 兩 although the factor of safety approach is overly con- lations for KVLCC2 AND KCS Models Using the PARNASSOS Code,’’ Pro-
servative for all grids. ceedings Gothenburg 2000 A Workshop on Numerical Ship Hydrodynamics,
Uncertainty estimates enable a quantitative measure of verifica- Gothenburg, Sweden.
关17兴 Ferziger, J. H., and Peric, M., 1996, Computational Methods for Fluid Dynam-
tion for analytical benchmarks. Figure 2共c, d兲 indicates that the ics, Springer-Verlag, New York.
present solutions are verified over the chosen range of grid size 关18兴 Eca, L., and Hoekstra, M., 2000, ‘‘On the Application of Verification Proce-
and time step. As expected, the largest levels of uncertainty are for dures in Computational Fluid Dynamics,’’ 2nd MARNET Workshop.
the coarsest grid size and time step where levels are (U k ,U k C ) 关19兴 Ebert, M. P., and Gorski, J. J., 2001, ‘‘A Verification and Validation Procedure
for Computational Fluid Dynamics Solutions,’’ NSWCCD-50-TR-2001/0006,
⫽(15%,7.5%). Hydromechanics Directorate Report, NSWC, Carderock Division, West Be-
Eca and Hoekstra 关18兴 also perform verification for the 2D thesda MD 20817-5700.

802 Õ Vol. 123, DECEMBER 2001 Transactions of the ASME

Downloaded 12 Jan 2011 to 59.162.23.4. Redistribution subject to ASME license or copyright; see http://www.asme.org/terms/Terms_Use.cfm