Attribution Non-Commercial (BY-NC)

38 views

Attribution Non-Commercial (BY-NC)

- 455984.fucak_fp
- FEA Basics very imp.ppt
- Rich Text Editor File
- CFD hvac
- M_Cely_0529
- Seakeeping prediction of KVLCC2 in head waves with RANS.pdf
- NAPA Product2008
- p19 Cala Unce Po
- Workshop Watermatex CFD
- Uncertainty Budget Table Template 16Jan2013
- Local_Scour_Free_Surface_Mesh.pdf
- CFD Computational Modeling of Nearfield to Farfield Plume Expansion AIAA-2007-5704.pdf
- syllabus cfd.pdf
- Workshop_watermatex_CFD.pdf
- Sim Multiphysics
- SGC2016 NortonStraw MattStraw Public
- Process Intensifier
- 3D-Free-Surface-Wave-Analysis-CFD-7.pdf
- hgODIJAIOF2B
- Uso Su Tre Casi Dei Modelli Cfd Per Prevedere l'Andamento Dei Fumi in Ambienti Chiusi Complessi

You are on page 1of 10

Comprehensive Approach to

Professor Mechanical Engineering and

Research Engineer

Fellow ASME

Verification and Validation of CFD

Robert V. Wilson Simulations—Part 1:

Assistant Research Engineer

Mem. ASME Methodology and Procedures

Iowa Institute Hydraulic Research,

The University of Iowa,

Part 1 of this two-part paper presents a comprehensive approach to verification and

Iowa City, IA 52242

validation methodology and procedures for CFD simulations from an already developed

CFD code applied without requiring availability of the source code for specified objec-

tives, geometry, conditions, and available benchmark information. Concepts, definitions,

and equations derived for simulation errors and uncertainties provide the overall math-

Hugh W. Coleman ematical framework. Verification is defined as a process for assessing simulation numeri-

Eminent Scholar in Propulsion,

cal uncertainty and, when conditions permit, estimating the sign and magnitude of the

Professor of Mechanical Engineering,

numerical error itself and the uncertainty in that error estimate. The approach for esti-

Propulsion Research Center,

mating errors and uncertainties includes (1) the option of treating the numerical error as

Mechanical and Aerospace Engineering

deterministic or stochastic, (2) the use of generalized Richardson extrapolation for J input

Department,

parameters, and (3) the concept of correction factors based on analytical benchmarks,

University of Alabama in Huntsville,

which provides a quantitative metric to determine proximity of the solutions to the

Huntsville, AL 35899

asymptotic range, accounts for the effects of higher-order terms, and are used for defining

Fellow ASME

and estimating errors and uncertainties. Validation is defined as a process for assessing

simulation modeling uncertainty by using benchmark experimental data and, when con-

ditions permit, estimating the sign and magnitude of the modeling error itself. The ap-

Eric G. Paterson proach properly takes into account the uncertainties in both the simulation and experi-

Associate Research Engineer,

mental data in assessing the level of validation. Interpretation of results of validation

Iowa Institute Hydraulic Research,

efforts both where the numerical error is treated as deterministic and stochastic are

The University of Iowa,

discussed. Part 2 provides an example for RANS simulations for a cargo/container ship

Iowa City, IA 52242

where issues with regard to practical application of the methodology and procedures and

Mem. ASME

interpretation of verification and validation results are discussed.

关DOI: 10.1115/1.1412235兴

uncertainty analysis 共Coleman and Steele 关7兴兲, which has been

Discussion and methodology for estimating errors and uncer-

standardized. Hopefully, CFD verification and validation proce-

tainties in computational fluid dynamics 共CFD兲 simulations have

dures and methodology can reach a similar level of maturity and

reached a certain level of maturity with recognition of importance

user variability can reach similar low levels, as for EFD. The work

through editorial policies 共Freitas 关1兴兲, increased attention and re-

is part of a larger program 共Rood 关8兴兲 for developing and imple-

cent progress on common terminology 共AIAA, 关2兴兲, advocacy and

menting a strategy for verification and validation of Reynolds-

detailed methodology 共Roache 关3兴兲, and numerous case studies

averaged Navier-Stokes 共RANS兲 ship hydrodynamics CFD codes.

共e.g. 关4兴兲. Progress has been accelerated in response to the urgent

The program includes complementary CFD and EFD towing-tank

need for achieving consensus on concepts, definitions, and useful

investigations and considers errors and uncertainties in both the

methodology and procedures, as CFD is applied to increasingly

simulations and the data in assessing the success of the verifica-

complex geometry and physics and integrated into the engineering

tion and validation efforts. The work also benefited from collabo-

design process. Such consensus is required to realize the goals of

ration with the 21st and 22nd International Towing Tank Resistance

simulation-based design and other uses of CFD such as simulating

Committees 共ITTC 关9,10兴兲. The procedures proposed in this paper

flows for which experiments are difficult 共e.g., full-scale Reynolds

were adopted on an interim basis by the 22nd ITTC and also were

numbers, hypersonic flows, off-design conditions兲. In spite of the

recommended and used at the recent Gothenburg 2000 Workshop

progress and urgency, the various viewpoints have not converged

on CFD in Ship Hydrodynamics 共Larsson et al. 关11兴兲.

and current approaches fall short of providing practical methodol-

The focus is on verification and validation methodology and

ogy and procedures for estimating errors and uncertainties in CFD

procedures for CFD simulations with an already developed CFD

simulations.

code applied without requiring availability of the source code for

The present work provides a pragmatic approach for estimating

specified objectives, geometry, conditions, and available bench-

errors and uncertainties in CFD simulations. Previous work on

mark information. The methodology and procedures were devel-

verification 共Stern et al. 关5兴兲 is extended and put on a more rigor-

oped considering RANS CFD codes, but should be applicable to a

ous foundation and combined with subsequent work on validation fairly broad range of codes such as boundary-element methods

共Coleman and Stern 关6兴兲 thereby providing a comprehensive and certain aspects of large-eddy and direct numerical simula-

framework for overall procedures and methodology. The philoso- tions. The present work differs in many respects from recent lit-

erature. The presentation is relatively succinct with intention for

Contributed by the Fluids Engineering Division for publication in the JOURNAL

OF FLUIDS ENGINEERING. Manuscript received by the Fluids Engineering Division

use for practical applications 共i.e., industrial CFD兲 for which nu-

November 4, 1999; revised manuscript received July 10, 2001. Associate Editor: merical errors and uncertainties cannot be considered negligible or

P. E. Raad. overlooked.

Journal of Fluids Engineering Copyright © 2001 by ASME DECEMBER 2001, Vol. 123 Õ 793

Downloaded 12 Jan 2011 to 59.162.23.4. Redistribution subject to ASME license or copyright; see http://www.asme.org/terms/Terms_Use.cfm

The definitions of errors and uncertainties and verification and ing to the error in ␦ * . When ␦ * is estimated, it can be used to

validation that are used in any approach need to be clearly stated. obtain a corrected value of the variable of interest.

The present and Roache 关3兴 definitions for errors and uncertainties Sources of errors and uncertainties in results from simulations

are consistent with those used for EFD. The AIAA 关2兴 definitions can be divided into two distinct sources: modeling and numerical.

are from an information theory perspective and differ from those Modeling errors and uncertainties are due to assumptions and ap-

used in EFD, but are not contradictory to the present definitions. proximations in the mathematical representation of the physical

The present concepts and definitions for verification and valida- problem 共such as geometry, mathematical equation, coordinate

tion are closely tied to the present definitions of errors and uncer- transformation, boundary conditions, turbulence models, etc.兲 and

tainties and equations derived for simulation errors and uncertain- incorporation of previous data 共such as fluid properties兲 into the

ties thereby providing the overall mathematical framework. The model. Numerical errors and uncertainties are due to numerical

Roache 关3兴 and AIAA 关2兴 definitions are broader, but not contra- solution of the mathematical equations 共such as discretization, ar-

dictory to the present definitions. The present approach includes tificial dissipation, incomplete iterative and grid convergence, lack

both the situations 共1兲 of estimating errors and the uncertainty of of conservation of mass, momentum, and energy, internal and ex-

those estimates and 共2兲 of estimating uncertainties only. Richard- ternal boundary noncontinuity, computer round-off, etc.兲. The

son extrapolation 共RE兲 is used for verification, which is not new; present work assumes that all correlations among errors are zero,

however, the present generalizations for J input parameters and which is doubtless not true in all cases, but the effects are assumed

concept of correction factors based on analytical benchmarks, negligible for the present analyses.

which provides a quantitative metric to determine proximity of the The simulation error ␦ S is defined as the difference between a

solutions to the asymptotic range, accounts for the effects of simulation result S and the truth T. In considering the develop-

higher-order terms, and are used for defining and estimating errors ment and execution of a CFD code, it can be postulated that ␦ S is

and uncertainties constitute a new approach. The use of quantita- comprised of the addition of modeling and numerical errors

tive estimates for errors and the use of uncertainties for those

estimates also constitute a new approach in verification and ␦ S ⫽S⫺T⫽ ␦ SM ⫹ ␦ SN (1)

validation. Support for this postulation is provided by using the model value

Part 1 of this two-part paper presents the verification and vali- M in definitions for modeling and numerical errors. The simula-

dation methodology and procedures. In Section 2, the overall veri- tion modeling error ␦ SM ⫽M ⫺T is defined as the difference be-

fication and validation methodology is presented by providing tween the true T and model M values while the simulation numeri-

concepts, definitions, and equations for the simulation numerical cal error ␦ SN ⫽S⫺M is defined as the difference between the

and modeling errors and uncertainties. In Section 3, detailed veri- simulation S and model M values. The simulation S and model M

fication procedures for estimation of various sub-components of values are obtained by numerical and exact solutions of the con-

the simulation numerical error and uncertainty are given. In Sec- tinuous equations used to model the truth, respectively. Since ex-

tion 4, validation procedures are given including a discussion of act solution of nonlinear equations is seldom possible, approxima-

the interpretation of validation results and use of corrected simu- tions are used to replace the continuous modeled equations with

lation results. Finally, conclusions are provided in Section 5. Part discrete ones that are solved algebraically with a CFD code to

2 provides an example for RANS simulations for a cargo/ yield the simulation value S.

container ship where issues with regard to practical application of The uncertainty equation corresponding to error equation 共1兲 is

the methodology and procedures and interpretation of verification

and validation results are discussed 共Wilson et al. 关12兴兲. Present U 2S ⫽U SM

2

⫹U SN

2

(2)

papers are based on Stern et al. 关13兴, which is sometimes refer- where U S is the uncertainty in the simulation and U SM and U SN

enced for additional details. However, presentation and expanded are the simulation modeling and numerical uncertainties.

discussions of verification procedures and implementation were For certain conditions, the numerical error ␦ SN can be consid-

improved based on nearly two years experience with present ap- ered as

proach, especially through ITTC community and Gothenburg

2000 Workshop on CFD in Ship Hydrodynamics. ␦ SN ⫽ ␦ SN

* ⫹ SN (3)

where ␦ SN* is an estimate of the sign and magnitude of ␦ SN and

2 Overall Verification and Validation Methodology SN is the error in that estimate 共and is estimated as an uncertainty

since only a range bounding its magnitude and not its sign can be

In Section 2.1, the overall verification and validation method- estimated兲. The corrected simulation value S C is defined by

ology is presented by providing key concepts, definitions, and

derivation of equations for the simulation error and uncertainty, as S C ⫽S⫺ ␦ SN

* (4)

sum and root-sum-square 共RSS兲 of simulation numerical and

modeling errors and uncertainties, respectively. The verification with error equation

and validation equations are derived in Sections 2.2 and 2.3, re- ␦ S C ⫽S C ⫺T⫽ ␦ SM ⫹ SN (5)

spectively, where subcomponents of the simulation numerical er-

ror are identified and an approach for assessing the simulation The uncertainty equation corresponding to error equation 共5兲 is

modeling uncertainty is presented.

U S2 C ⫽U SM

2

⫹U S2 C N (6)

2.1 Concepts and Definitions. Accuracy indicates the

closeness of agreement between a simulation/experimental value where U S C is the uncertainty in the corrected simulation and U S C N

of a quantity and its true value. Error ␦ is the difference between is the uncertainty estimate for SN .

a simulation value or an experimental value and the truth. Accu- Debate on verification and validation has included discussion

racy increases as error approaches zero. The true values of on whether errors such as ␦ SN are deterministic or stochastic, and

simulation/experimental quantities are rarely known. Thus, errors thus how they should be treated in uncertainty analysis was un-

must be estimated. An uncertainty U is an estimate of an error clear. In the ‘‘corrected’’ approach given by Eqs. 共3兲–共6兲, a deter-

such that the interval ⫾U contains the true value of ␦ 95 times ministic estimate ␦ SN

* of ␦ SN and consideration of the error SN in

out of 100. An uncertainty interval thus indicates the range of that estimate are used. The approach is analogous to that in EFD

likely magnitudes of ␦ but no information about its sign. when an asymmetric systematic uncertainty is ‘‘zero-centered’’ by

For simulations, under certain conditions, errors can be esti- inclusion of a model for the systematic error in the data reduction

mated including both sign and magnitude 共referred to as an error equation and then the uncertainty considered is that associated

estimate ␦ * 兲. Then, the uncertainty considered is that correspond- with the model 共Coleman and Steele 关7兴兲. In the ‘‘uncorrected’’

Downloaded 12 Jan 2011 to 59.162.23.4. Redistribution subject to ASME license or copyright; see http://www.asme.org/terms/Terms_Use.cfm

approach given by Eqs. 共1兲–共2兲, any particular ␦ SN is considered

as a single realization from some parent population of ␦ SN ’s and

the uncertainty U SN is interpreted accordingly in analogy to the

estimation of uncertainties in EFD 共with a similar argument for

SN and U S C N 兲. Oberkamp and Trucano 关14兴 have criticized Cole-

man and Stern 关6兴 for treating U SN statistically; however, the

present approach is well justified both conceptually and math-

ematically for reasons just given.

The overall CFD verification and validation procedures can be

conveniently grouped in four consecutive steps. The first step is

preparation, which involves selection of the CFD code and speci-

fication of objectives, geometry, conditions, and available bench-

mark information. The objectives might be prediction of certain

variables at certain levels of validation 共e.g., programmatic vali-

dation requirements U reqd 兲. The variables can either be integral

共e.g., resistance兲 or point 共e.g., mean velocities and turbulent Rey-

nolds stresses兲 values and the programmatic validation require-

ments may be different for each variable. The second and third

Fig. 1 Definition of comparison error.

steps are verification and validation, which are described in Sec-

tions 2.2 and 2.3. The fourth step is documentation, which is

detailed presentation of the CFD code 共equations, initial and

boundary conditions, modeling, and numerical methods兲, objec- mark since it is equal, as shown by Eq. 共10兲, to the truth plus

tives, geometry, conditions, verification, validation, and analysis. simulation modeling error and presumable small error SN in the

2.2 Verification. Verification is defined as a process for as- estimate of the numerical error ␦ SN

* .

sessing simulation numerical uncertainty U SN and, when condi- 2.3 Validation. Validation is defined as a process for assess-

tions permit, estimating the sign and magnitude ␦ SN* of the simu- ing simulation modeling uncertainty U SM by using benchmark

lation numerical error itself and the uncertainty in that error experimental data and, when conditions permit, estimating the

estimate. For many CFD codes, the most important numerical sign and magnitude of the modeling error ␦ SM itself. Thus, the

errors and uncertainties are due to use of iterative solution meth- errors and uncertainties in the experimental data must be consid-

ods and specification of various input parameters such as spatial ered in addition to the numerical errors and uncertainties dis-

and time step sizes and other parameters 共e.g., artificial dissipa- cussed in Section 3. Approaches to estimating experimental un-

tion兲. The errors and uncertainties are highly dependent on the certainties are presented and discussed by Coleman and Steele 关7兴.

specific application 共geometry and conditions兲. The validation methodology of Coleman and Stern 关6兴 which

The errors due to specification of input parameters are decom- properly takes into account the uncertainties in both the simula-

posed into error contributions from iteration number ␦ I , grid size tion and the experimental data is discussed in this section for both

␦ G , time step ␦ T , and other parameters ␦ P , which gives the approaches of treating the numerical error as stochastic and as

following expressions for the simulation numerical error and un- deterministic.

certainty The validation comparison is shown in Fig. 1. The experimen-

J

tally determined r-value of the (X i ,r i ) data point is D and simu-

lated r-value is S. Recall from Eq. 共1兲 that the simulation error ␦ S

␦ SN ⫽ ␦ I ⫹ ␦ G ⫹ ␦ T ⫹ ␦ P ⫽ ␦ I ⫹ 兺␦

j⫽1

j (7) is the difference between S and the truth T. Similarly, the error ␦ D

in the data is the difference between D and the truth T, so setting

J the simulation and experimental truths equal results in

2

U SN ⫽U I2 ⫹U G

2

⫹U T2 ⫹U 2P ⫽U I2 ⫹ 兺U

j⫽1

2

j (8) D⫺ ␦ D ⫽S⫺ ␦ S (13)

The comparison error E is defined as the difference of D and S

Similarly, error estimates ␦ * can be decomposed as

E⫽D⫺S⫽ ␦ D ⫺ ␦ S ⫽ ␦ D ⫺ 共 ␦ SM A ⫹ ␦ S PD ⫹ ␦ SN 兲 (14)

J

with ␦ SM decomposed into the sum of ␦ S PD , error from the use of

␦ SN

* ⫽ ␦ I* ⫹ 兺 ␦*

j⫽1

j (9) previous data such as fluid properties, and ␦ SM A , error from mod-

eling assumptions. Thus E is the resultant of all the errors asso-

which gives the following expressions for the corrected simulation ciated both with the experimental data and with the simulation.

and corrected simulation numerical uncertainty For the approach in which no estimate ␦ SN * of the sign and mag-

nitude of ␦ SN is made, all of these errors are estimated with un-

冉

S C ⫽S⫺ ␦ I* ⫹

J

兺 ␦*

j⫽1

j 冊 ⫽T⫹ ␦ SM ⫹ SN (10)

certainties.

If (X i ,r i ), and S share no common error sources, then the un-

certainty U E in the comparison error can be expressed as

U S2 C N ⫽U I2C ⫹ 兺

j⫽1

J

U 2j C (11) U E2 ⫽ 冉 冊 冉 冊

E

D

2

2

UD ⫹

E

S

2

⫹U 2S ⫽U D

2

⫹U 2S (15)

or

Verification is based on equation 共10兲, which is put in the form

U E2 ⫽U D

2

⫹U SM

2

A ⫹U S PD ⫹U SN

2 2

冉 冊

(16)

J

S⫽S C ⫹ ␦ I* ⫹ 兺 ␦*

j⫽1

j (12)

Ideally, one would postulate that if the absolute value of E is

less than its uncertainty U E , then validation is achieved 共i.e., E is

‘‘zero’’ considering the resolution imposed by the ‘‘noise level’’

Equation 共12兲 expresses S as the corrected simulation value S C U E 兲. In reality, there is no known approach that gives an estimate

plus numerical errors. S C is also referred to as a numerical bench- of U SM A , so U E cannot be estimated. That leaves a more stringent

Downloaded 12 Jan 2011 to 59.162.23.4. Redistribution subject to ASME license or copyright; see http://www.asme.org/terms/Terms_Use.cfm

validation test as the practical alternative. If the validation uncer- 3 Verification Procedures

tainty U V is defined as the combination of all uncertainties that we

In Section 2, the simulation numerical error and uncertainty

know how to estimate 共i.e., all but U SM A 兲, then

were decomposed into contributions from iteration number, grid

size, time step, and other parameters in Eqs. 共7兲 and 共8兲. In this

U V2 ⫽U E2 ⫺U SM

2

A ⫽U D ⫹U STE

2 2

(17) section, detailed verification procedures are given for estimation

of these contributions through convergence studies 共Section 3.1兲.

2

where U STE ⫽U S2 PD ⫹U SN

2

is the total estimated simulation uncer- Iterative 共Section 3.2兲 and parameter 共Sections 3.3–3.5兲 conver-

tainty, as shown in Fig. 1. gence studies are conducted using multiple solutions with system-

If 兩E兩 is less than the validation uncertainty U V , the combina- atic parameter refinement to estimate numerical errors and uncer-

tion of all the errors in D and S is smaller than the estimated tainties. Three convergence conditions are possible: 共i兲 monotonic

validation uncertainty and validation has been achieved at the U V convergence; 共ii兲 oscillatory convergence; and 共iii兲 divergence and

level. U V is the key metric in the validation process. U V is the are described in Sections 3.3, 3.4, and 3.5, respectively. For con-

validation ‘‘noise level’’ imposed by the uncertainties inherent in dition 共i兲, as already mentioned, errors and uncertainties are esti-

the data, the numerical solution, and the previous experimental mated using generalized RE. For condition 共ii兲, uncertainties are

data used in the simulation model. It can be argued that one can- estimated simply by attempting to bound error based on oscilla-

not discriminate once 兩E兩 is less than this; that is, as long as 兩E兩 is tion maximums and minimums. For condition 共iii兲, errors and un-

less than this, one cannot evaluate the effectiveness of proposed certainties cannot be estimated. As discussed below and later in

model ‘‘improvements.’’ On the other hand, if 兩 E 兩 ⰇU V one could Section 5, there are many issues in estimating errors and uncer-

argue that probably E⬇ ␦ SM A . tainties for practical applications.

Oberkamp and Trucano 关14兴 have criticized Coleman and Stern

关6兴 for fact that U V excludes U SM A . As already acknowledged, 3.1 Convergence Studies. Iterative and parameter conver-

there is no known way for directly estimating U SM A . However, gence studies are conducted using multiple 共m兲 solutions and sys-

the present approach does provide a more stringent validation tematic parameter refinement by varying the kth input parameter

metric U V which sets the level that validation can be achieved as ⌬x k while holding all other parameters constant. The present work

the root sum square of the experimental U D and the total esti- assumes input parameters can be expressed such that the finest

mated simulation U STE uncertainties. Additionally, under certain resolution corresponds to the limit of infinitely small parameter

conditions, the simulation modeling error ␦ SM A itself can be esti- values. Many common input parameters are of this form, e.g., grid

mated, as further discussed in Section 4. Consideration of Eq. 共17兲 spacing, time step, and artificial dissipation. Additionally, a uni-

shows that 共1兲 the more uncertain the data 共greater U D 兲 and/or 共2兲 form parameter refinement ratio r k ⫽⌬x k 2 /⌬x k 1 ⫽⌬x k 3 /⌬x k 2

the more inaccurate the code 共greater U STE 兲, the easier it is to ⫽⌬x k m /⌬x k m⫺1 between solutions is assumed for presentation

validate a code, since the greater the uncertainties in the data and purposes, but not required as discussed later.

code predictions, the greater the noise level U V . Both Roache 关3兴 Careful consideration should be given to selection of uniform

and Oberkamp and Trucano 关14兴 have criticized Coleman and parameter refinement ratio. The most appropriate values for indus-

Stern 关6兴 for this fact. However, if the value of U V is greater than trial CFD are not yet fully established. Small values 共i.e., very

that designated as necessary in a research/design/development close to one兲 are undesirable since solution changes will be small

program, the required level of validation could not be achieved and sensitivity to input parameter may be difficult to identify com-

without improvement in the quality of the data, the code, or both. pared to iterative errors. Large values alleviate this problem; how-

Also, if U SN and U S PD are not estimated, but 兩E兩 is less than U D , ever, they also may be undesirable since the finest step size may

then a type of validation can be argued to have been achieved, but be prohibitively small 共i.e., require many steps兲 if the coarsest step

clearly as shown by the present methodology, at an unknown size is designed for sufficient resolution such that similar physics

level. are resolved for all m solutions. Also, similarly as for small val-

If the ‘‘corrected’’ approach of Eqs. 共3兲–共6兲 is used, then the ues, solution changes for the finest step size may be difficult to

equations, equivalent to Eqs. 共14兲 and 共17兲 are identify compared to iterative errors since iterative convergence is

more difficult for small step size. Another issue is that for param-

E C ⫽D⫺S C ⫽ ␦ D ⫺ 共 ␦ SM A ⫹ ␦ S PD ⫹ SN 兲 (18) eter refinement ratio other than r k ⫽2, interpolation to a common

location is required to compute solution changes, which intro-

for the corrected comparison error and duces interpolation errors. Roache 关3兴 discusses methods for

evaluating interpolation errors. However, for industrial CFD, r k

U V2 C ⫽U E2 C ⫺U SM

2

A ⫽U D ⫽U D ⫹U S C TE

2 2 2

(19) ⫽2 may often be too large. A good alternative may be r k ⫽&, as

it provides fairly large parameter refinement ratio and at least

for the corrected validation uncertainty where U S2 C TE ⫽U S2 PD enables prolongation of the coarse-parameter solution as an initial

guess for the fine-parameter solution.

⫹U S2 C N is the total estimated corrected simulation uncertainty,

Equation 共12兲 is written for the kth parameter and mth solution

also shown in Fig. 1. Note that S C and E C can be either larger or as

smaller than their counterparts S and E, but U E C and U V C should

f

兺

be smaller than U E and U V , respectively, since U S C N should be

S k m ⫽S C ⫹ ␦ I*k ⫹ ␦ *

km⫹ ␦ *j m (20)

smaller than U SN . m j⫽1,j⫽k

If there is a programmatic validation requirement, there is an-

other uncertainty U reqd that must be considered since validation is Iterative convergence must be assessed and S k m corrected for it-

required at that uncertainty level or below. Interpretation of the erative errors prior to evaluation of parameter convergence since

meaning of the relative magnitudes of E 共or E C 兲, U reqd and U V the level of iterative convergence may not be the same for all m

共or U V C 兲 and of the implications on the possibility of estimating solutions used in the parameter convergence studies. Equation

␦ SM A are discussed in Section 4. Additional discussion is provided 共20兲 shows that iterative errors ␦ I*k must be accurately estimated

m

in Coleman and Stern 关6兴 on: estimating U S PD ; estimating U D for or negligible in comparison to ␦ * k m for accurate convergence stud-

the data point (X i ,r i ), including both the experimental uncer-

tainty in r i and the additional uncertainties in r i arising from the ies and that they should be considered within the context of con-

experimental uncertainties in the measurements of the n indepen- vergence studies for each input parameter. Methods for estimating

dent variables (X j ) i in X i ; and for validation of a CFD code, U I or ␦ I* and U I C are described in Section 3.2.2.

multiple codes and/or models, and prediction of trends. With ␦ I*k evaluated, S k m is corrected for iterative errors as

m

Downloaded 12 Jan 2011 to 59.162.23.4. Redistribution subject to ASME license or copyright; see http://www.asme.org/terms/Terms_Use.cfm

J gence study on the amplitude distribution of the Fourier modes. In

Ŝ k m ⫽S k m ⫺ ␦ I*k ⫽S C ⫹ ␦ k*m ⫹

m

兺

j⫽1,j⫽k

␦ *j m (21) principle, this approach would remove the problem of ill-

conditioning of the convergence ratio, R k .

Ŝ k m can be calculated for both integral 共e.g., resistance coeffi- 3.2 Iterative Convergence. The number of order magni-

cients兲 and point 共e.g., surface pressure, wall-shear stress, and tude drop and final level of solution residual 共or residual imbal-

velocity兲 variables. Ŝ k m can be presented as an absolute quantity ance兲 can be used to determine stopping criteria for iterative so-

共i.e., non-normalized兲 or normalized with the solution as a per- lution techniques. Iterative convergence to machine zero is

centage change; however, if the solution value is small, a more desirable, but for complex geometry and conditions it is often not

appropriate normalization may be the range of the solution. possible. Three or four orders of magnitude drop in solution re-

Convergence studies require a minimum of m⫽3 solutions to sidual to a level of 10⫺4 is more likely for these cases. Methods

evaluate convergence with respect to input parameter. Note that for estimation of iterative errors and uncertainties can be based on

m⫽2 is inadequate, as it only indicates sensitivity and not con- graphical, as discussed below, or theoretical approaches and are

vergence, and that m⬎3 may be required. Consider the situation dependent on the type of iterative convergence: 共a兲 oscillatory; 共b兲

convergent; or 共c兲 mixed oscillatory/convergent.

for 3 solutions corresponding to fine Ŝ k l , medium Ŝ k 2 , and coarse

For oscillatory iterative convergence 共a兲, the deviation of the

Ŝ k 3 values for the kth input parameter. Solution changes for variable from its mean value provides estimates of the iterative

medium-fine and coarse-medium solutions and their ratio R k are uncertainty based on the range of the maximum S U and minimum

defined by S L values

k 21⫽Ŝ k 2 ⫺Ŝ k 1

U I⫽ 冏 1

共 S ⫺S L 兲

2 U

冏 (24)

k 32⫽Ŝ k 3 ⫺Ŝ k 2 (22)

For convergent iterative convergence 共b兲, a curve-fit of an ex-

R k ⫽ k 21 / k 32 ponential function can be used to estimate U I or ␦ I* and U I C as

the difference between the value and the exponential function

Three convergence conditions are possible: from a curve fit for large iteration number CF ⬁

共 i 兲 Monotonic convergence: 0⬍R k ⬍1 U I ⫽ 兩 S⫺CF ⬁ 兩

m

amplitude of the solution envelope decreases as the iteration num-

For monotonic convergence 共i兲, generalized RE is used to esti-

ber increases, the solution envelope is used to define the maxi-

mate U k or ␦ * k and U k C . Methods for estimating errors and un- mum S U and minimum S L values in the Ith iteration, and to esti-

certainties for condition 共i兲 are described in Section 3.3. mate U I or ␦ I* and U I C

For oscillatory convergence 共ii兲, the solutions exhibit oscilla-

tions, which may be erroneously identified as condition 共i兲 or 共iii兲.

This is apparent if one considers evaluating convergence condi-

tion from three points on a sinusoidal curve 共Coleman et al. 关15兴兲.

U I⫽ 冏 1

共 S ⫺S L 兲

2 U

冏 (26)

Depending on where the three points fall on the curve, the condi- 1

tion could be incorrectly diagnosed as either monotonic conver- ␦ I*k ⫽S⫺ 共 S U ⫺S L 兲 ,U I C ⫽0

m 2

gence or divergence. Methods discussed here for estimating un-

certainties U k for condition 共ii兲 require more than m⫽3 solutions An increase in the amplitude of the solution envelope as the

and are described in Section 3.4. iteration number increases indicates that the solution is divergent.

For divergence 共iii兲, the solutions diverge and errors and uncer- Estimates of the iterative error based on theoretical approaches

tainties cannot be estimated. Additional remarks are given in Sec- are presented in Ferziger and Peric 关17兴 and involve estimation of

tion 3.5. the principal eigenvalue of the iteration matrix. The approach is

Determination of the convergence ratio R k for point variables relatively straightforward when the eigenvalue is real and the so-

can be problematic since solution changes k 21 and k 32 can both lution is convergent. For cases in which the principal eigenvalue is

go to zero 共e.g., in regions where the solution contains an inflec- complex and the solution is oscillatory or mixed, the estimation is

tion point兲. In this case, the ratio becomes ill conditioned. How- not as straightforward and additional assumptions are required.

ever, the convergence ratio can be used in regions where the so-

lution changes are both non-zero 共e.g., local solution maximums 3.3 Monotonic Convergence: Generalized Richardson Ex-

or minimums兲. Another approach is to use a global convergence trapolation. For monotonic convergence, i.e., condition 共i兲 in

ratio R k , which overcomes ill conditioning, based on the L2 norm Eq. 共23兲, generalized RE is used to estimate U k or ␦ *k and U k C .

of the solution changes, i.e., 具 R k 典 ⫽ 储 k 21储 2 / 储 k 32储 2 . 具 典 is used to RE is generalized for J input parameters and concept of correction

denote an averaged value and 储 储 2 ⫽ 关 兺 i⫽1 N

i2 兴 1/2 denotes the L2 factors based on analytical benchmarks is introduced. More de-

norm of solution change over the N points in the region of inter- tailed derivations are provided by Stern et al. 关13兴.

est. Caution should be exercised when defining the convergence As already mentioned, since Stern et al. 关13兴 there has been

ratio from the ratio of the L2 norm of solution changes because nearly two years experience with present approach, especially

the oscillatory condition (R k ⬍1) cannot be diagnosed since 具 R k 典 through ITTC community and Gothenburg 2000 Workshop on

will always be greater than zero. Local values of R k at solution CFD in Ship Hydrodynamics. In particular, detailed verification

maximums or minimums should also be examined to confirm the procedures have been the focus of attention 共Eca and Hoekstra

convergence condition based on an L2 norm definition. An alter- 关18兴; Ebert and Gorski 关19兴兲. After some background for general-

nate approach suggested by Hoekstra et al. 关16兴 is to transform the ized RE is given, two approaches for estimating errors and uncer-

spatial profile to wave number space and to perform a conver- tainties are presented and are based on 共i兲 correction factors pro-

posed in the current paper and 共ii兲 factor of safety approach

1

As discussed in the text that follows, 0⬍R k ⬍1 and R k ⬎1 may also occur for proposed by Roache 共1998兲. Finally, a discussion of fundamental

the oscillatory condition. and practical issues for verification is provided.

Downloaded 12 Jan 2011 to 59.162.23.4. Redistribution subject to ASME license or copyright; see http://www.asme.org/terms/Terms_Use.cfm

Background for Generalized RE. Generalized RE begins with medium input parameters r k 32 can be used to estimate ␦ * k 1 from

Eq. 共21兲. The error terms on the right-hand side of Eq. 共21兲 are of Eq. 共29兲, provided that Eq. 共30兲 for estimating order of accuracy is

known form 共i.e., power series expansion with integer powers of modified as

⌬x k 兲 based on analysis of the modified and numerical error equa-

tions which is written below as a finite sum 共i.e., error estimate兲 ln共 k 32 / k 21兲 1 p p

and for the kth parameter and mth solution p k⫽ ⫹ 关 ln共 r k k ⫺1 兲 ⫺ln共 r k k ⫺1 兲兴

ln共 r k 21兲 ln共 r k 21兲 32 21

n

兺 共 ⌬x

i兲

p共

␦ *k m ⫽ km兲 k g 共ki 兲 (27) (31)

i⫽1

For situations when r k 21⫽r k 32, Eq. 共31兲 is a transcendental equa-

n⫽number of terms retained in the power series, powers p (ki ) cor-

tion implicitly defining p k and must be solved iteratively. If r k 21

respond to order of accuracy 共for the ith term兲, and g (ki ) are re-

⫽r k 32, Eq. 共31兲 degenerates to Eq. 共30兲.

ferred to as ‘‘grid’’ functions which are a function of various or-

ders and combinations of derivatives of S with respect to x k . It is Estimating Errors and Uncertainties Using Generalized RE

assumed that the power series in Eq. 共27兲 is convergent 共i.e., the With Correction Factors. Results from the numerical solution of

finite sum convergence to the infinite series value as more terms the one-dimensional 共1D兲 wave and two-dimensional 共2D兲

are included兲. Substituting Eq. 共27兲 into Eq. 共21兲 results in Laplace equation analytical benchmarks show that Eq. 共29兲 has

n J the correct form, but the order of accuracy is poorly estimated by

兺 兺

共i兲

Ŝ k m ⫽S C ⫹ 共 ⌬x k m 兲 p k g 共ki 兲 ⫹ ␦ *j m (28) Eq. 共30兲 except in the asymptotic range. Analysis of the results

i⫽1 j⫽1,j⫽k suggests the concept of correction factors, which provide a quan-

titative metric to determine proximity of the solutions to the

Subtraction of multiple solutions where input parameter ⌬x k is asymptotic range, account for the effects of higher-order terms,

uniformly refined eliminates the ␦ *j m terms in Eq. 共28兲 since ␦ *j m is and are used for defining and estimating errors and uncertainties.

independent of ⌬x k and provides equations for S C , p (ki ) , and g (ki ) . Details are provided in Appendix A.

This assumes p (ki ) and g (ki ) are also independent of ⌬x k . Since Multiplication of Eq. 共29兲 by a correction factor C k provides an

each term 共i兲 contains 2 unknowns, m⫽2n⫹1 solutions are re- estimate for ␦ *

k 1 accounting for the effects of higher-order terms

冉 冊

quired to estimate the numerical benchmark S C and the first n

terms in the expansion in Eq. 共28兲 共i.e., for n⫽1, m⫽3 and for k 21

n⫽2, m⫽5, etc兲. The accuracy of the estimates depends on how ␦ *k 1 ⫽C k ␦ RE

* ⫽C k p (32)

k 1 r k k ⫺1

many terms are retained in Eq. 共27兲, the magnitude 共importance兲

of the higher-order terms, and the validity of the assumption that If solutions are in the asymptotic range, correction of Eq. 共29兲 is

p (ki ) and g (ki ) are independent of ⌬x k . For sufficiently small ⌬x k , not required 关i.e., C k ⫽1 so that Eqs. 共29兲 and 共32兲 are equivalent兴.

the solutions are in the asymptotic range such that higher-order For solutions outside the asymptotic range, C k ⬍1 or C k ⬎1 indi-

terms are negligible and the assumption that p (ki ) and g (ki ) are in- cates that the leading-order term over predicts 共higher-order terms

dependent of ⌬x k is valid. However, achieving the asymptotic net negative兲 or under predicts 共higher-order terms net positive兲

range for practical geometry and conditions is usually not possible the error, respectively. The estimate given by Eq. 共32兲 includes

and m⬎3 is undesirable from a resources point of view; therefore, both sign and magnitude and is used to estimate U k or ␦ * k and U k C

methods are needed to account for effects of higher-order terms depending on how close the solutions are to the asymptotic range

for practical application of RE. Additionally, methods may be 共i.e., how close C k is to 1兲 and one’s confidence in Eq. 共32兲. There

needed to account for possible dependence of p (ki ) and g (ki ) on are many reasons for lack of confidence, especially for complex

⌬x k , although not addressed herein. Usually ␦ k* is estimated for three-dimensional flows.

the finest value of the input parameter, i.e., ␦ * k ⫽ ␦ k*1 corresponding

For C k sufficiently less than or greater than 1 and lacking con-

to the finest solution S k 1 . fidence, U k is estimated, but not ␦ * k and U k c . Equation 共32兲 is

With three solutions (m⫽3), only the leading-order term of used to estimate the uncertainty by bounding the error ␦ * k by the

Eq. 共27兲 can be estimated. Solution of the three equations for S C , sum of the absolute value of the corrected estimate from RE and

p (ki ) , and g (ki ) yields estimates for the error ␦ * the absolute value of the amount of the correction

k 1 and order-of-

accuracy p k U k ⫽ 兩 C k ␦ RE

* 兩 ⫹ 兩 共 1⫺C k 兲 ␦ RE

k

* 兩

k

(33)

1 1

k 21

␦ *k 1 ⫽ ␦ RE

* ⫽ p (29) For C k sufficiently close to 1 and having confidence, ␦ * k and

k r k k ⫺1 U k c are estimated. Equation 共32兲 is used to estimate the error ␦ *

k ,

1

ln共 k 32 / k 21兲 which can then also be used in the calculation of S c 关in Eq. 共10兲兴.

p k⫽ (30) The uncertainty in the error estimate is based on the amount of the

ln共 r k 兲 correction

Solving for the first-order term is relatively easy since evaluation U k C ⫽ 兩 共 1⫺C k 兲 ␦ RE

* 兩 (34)

of Eqs. 共29兲 and 共30兲 only requires that the m⫽3 solutions are k 1

Note that in the limit of the asymptotic range, C k ⫽1, ␦ *

k ⫽␦*

asymptotic range and Eqs. 共29兲 and 共30兲 are inaccurate. With so- k1

*

⫽5), more complicated expressions can be derived to estimate

1

Two definitions for the correction factor were developed. The

the first two terms of the power series expansion. However, their first is based on solution of Eq. 共32兲 for C k with ␦ RE

* based on

range of applicability is more restrictive since all five solutions k 1

must be both monotonically convergent and sufficiently close to Eq. 共29兲 but replacing p k with the improved estimate p k est

the asymptotic range for the expressions to be used. p

As previously mentioned, solutions from three values of input r k k ⫺1

parameter where the refinement ratio between the medium and C k⫽ p (35)

fine input parameters r k 21 is not equal to that between coarse and r k k est ⫺1

Downloaded 12 Jan 2011 to 59.162.23.4. Redistribution subject to ASME license or copyright; see http://www.asme.org/terms/Terms_Use.cfm

Similarly, the second is based on a two-term estimate of the power been used to address some of these fundamental issues while oth-

series which is used to estimate ␦ RE* where p k and q k are re-

k

ers need further research. Although both correction factor and

1

factor of safety approaches were presented, the authors advocate

placed with p k est and q k est the use of former. Results from the numerical solution of analytic

q p p p benchmarks show that the factor of safety approach is overly con-

共 k 23 / k 12⫺r k k est 兲共 r k k ⫺1 兲 共 k 23 / k 12⫺r k k est 兲共 r k k ⫺1 兲 servative, especially when the solutions approach the asymptotic

C k⫽ p q p ⫹ p q q range 共Appendix A兲. This is in contrast to the variable correction

共 r k k est ⫺r k k est 兲共 r k k est ⫺1 兲 共 r k k est ⫺r k k est 兲共 r k k est ⫺1 兲

factor approach proposed in Eqs. 共33兲 and 共34兲, where the uncer-

(36) tainty in the error estimate correctly goes to zero as the asymptotic

p k est and q k est are estimates for limiting orders of accuracy of the range is approached because C k →1. Admittedly, others have rec-

ommended the factor of safety approach, e.g., Eca and Hoekstra

first and second terms of the error expansion equation 共27兲 as

关18兴, although examination of their results as with our own analy-

spacing size goes to zero and the asymptotic range is reached.

sis indicates that such estimates are overly conservative.

Equation 共35兲 roughly accounts for the effects of higher-order

For practical applications, especially complex flows with rela-

terms by replacing p k with p k est thereby providing an improved

tively coarse grids, solutions may be far from asymptotic range

single-term estimate. Equation 共36兲 more rigorously accounts for such that some variables are convergent while others are oscilla-

higher-order terms since it is derived from the two-term estimate tory or even divergent. Order of accuracy and therefore correction

with first and second term order of accuracy p (k1 ) and p (k2 ) replaced factors and factors of safety may display large variability indicat-

by p k est and q k est . Equation 共36兲 simplifies to Eq. 共35兲 in the limit ing the need for finer grids. Clearly, more than 3 grids are required

of the asymptotic range. Both correction factors only require so- to estimate errors and uncertainties for such cases. Eca and Hoek-

lutions for three parameter values. The estimated values p k est and stra 关18兴 suggest a least-squares approach to estimate the error by

q k est can be based either on the assumed theoretical order of ac- computing the three unknown parameters from RE when more

than three solutions are available. The behavior of the asymptotic

curacy p k th and q k th or solutions for simplified geometry and con- range was successfully demonstrated for simpler analytical bench-

ditions. In either case, preferably including the effects of grid marks in Appendix A. However, the existence and behavior of the

stretching. asymptotic range for practical problems has not been demon-

In Appendix A, exact 共A兲 and numerical 共S兲 solutions are used strated due to lack of sufficiently refined grids, number of solu-

to compare the true simulation error (A⫺S) to 共i兲 an uncorrected tions to assess variability, and available resources, among other

three-grid error estimate using Eq. 共29兲 and 共ii兲 corrected esti- issues. Another practical issue involves selecting and maintaining

mates based on Eq. 共32兲 with correction factor defined by Eq. 共35兲 appropriate parameter refinement ratio and resources for obtaining

or 共36兲. Correction of error estimates with both definitions of C k solutions with sufficient parameter refinement as well as number

results in improved error estimates. Also, uncertainty estimates of solutions. Lastly, interpretation of results is an issue since, as

using Eq. 共33兲 with correction factor defined by Eq. 共35兲 or 共36兲 already mentioned, there is limited experience and no known

are shown to bound the true simulation error (A⫺S), while un- solutions for practical applications in the asymptotic range for

certainty estimates using Eq. 共34兲 are shown to bound the differ- guidance.

ence between the corrected solution and the truth (S c -T). Addi- The present verification procedures represent the most rational

tional testing of expressions for C k given by Eqs. 共35兲 and 共36兲 is approach presently known. However, alternative strategies for in-

needed and development of improved expressions within the pro- cluding effects of higher-order terms may be just as viable, e.g.,

posed general framework is certainly possible. treatment of the power series exponents as known integers as

Estimating Uncertainties Using Generalized RE With Factors proposed by Oberkampf and investigated by Eca and Hoekstra

of Safety. In Roache 关3兴, a GCI approach is proposed where a 关18兴. Once available, improved verification procedures can be eas-

standard three-grid error estimate from RE is multiplied by a fac- ily incorporated into the present overall verification and validation

tor of safety F S to bound the simulation error methodology. These issues are discussed further in Section 5 Con-

clusions and Recommendations and in Part 2 共Wilson et al. 关12兴兲.

U k ⫽F S 兩 ␦ RE

* 兩

k

(37) 3.4 Oscillatory Convergence. For oscillatory convergence,

1

i.e., condition 共ii兲 in Eq. 共23兲, uncertainties can be estimated, but

Note that Eq. 共37兲 with factor of safety differs significantly from not the signs and magnitudes of the errors. Uncertainties are esti-

Eq. 共34兲. Herein C k ⫽C k (,r k ,p k ,p k est ,q k est ), in contrast to Eq. mated based on determination of the upper (S U ) and lower (S L )

共37兲 where C k is a constant referred to as a factor of safety F S . bounds of solution oscillation, which requires more than m⫽3

The exact value for factor of safety is somewhat ambiguous and solutions. The estimate of uncertainty is based on half the solution

Roache 关3兴 recommends 1.25 for careful grid studies and 3 for range

cases in which only two grids are used.

Although not proposed in Roache 关3兴, the factor of safety ap- 1

U k ⫽ 共 S U ⫺S L 兲 (39)

proach can be used for situations where the solution is corrected 2

with an error estimate from RE. Equation 共29兲 is used to estimate

␦ k* and the uncertainty in that error estimate is given by 3.5 Divergence. For divergence, i.e., condition 共iii兲 in Eq.

共23兲, errors and or uncertainties can not be estimated. The prepa-

U k C ⫽ 共 F S ⫺1 兲 兩 ␦ RE

* 兩 (38) ration and verification steps must be reconsidered. Improvements

k 1 in iterative convergence, parameter specification 共e.g., grid qual-

With this approach, a fixed percentage of a three-grid error ity兲, and/or CFD code may be required to achieve converging or

oscillatory conditions.

estimate 共e.g., 25% ␦ RE

* for F S ⫽1.25兲 is used to define the un-

k 1

certainty of the error estimate regardless of how close solutions 4 Validation Procedures

are to the asymptotic range.

In Section 2, an approach for assessing the simulation modeling

Discussion of Fundamental and Practical Issues. Fundamen- uncertainty was presented where for successful validation, the

tal and practical issues for verification are discussed in this sec- comparison error, E is less than the validation uncertainty, U V

tion. Fundamental issues include convergence of power series given by Eqs. 共17兲 and 共19兲 for uncorrected and corrected solu-

equation 共27兲, assumptions that p (ki ) and g (ki ) are independent of tions, respectively. In this section, validation procedures are pre-

⌬x k , and estimating p k est . Solution of analytical benchmarks has sented through discussions in Section 4.1 on interpretation of

Downloaded 12 Jan 2011 to 59.162.23.4. Redistribution subject to ASME license or copyright; see http://www.asme.org/terms/Terms_Use.cfm

validation results and in Section 4.2 on use of corrected simula- 5 Conclusions

tion results. As previously mentioned, Coleman and Stern 关6兴 pro-

The present comprehensive approach to verification and valida-

vide additional discussion on validation procedures.

tion methodology and procedures sets forth concepts, definitions,

and equations derived for simulation errors and uncertainties,

4.1 Interpretation of the Results of a Validation Effort. which provide a well-founded mathematical framework. The ap-

First, consider the approach in which the simulation numerical proach should have applicability to a fairly broad range of CFD

error is taken to be stochastic and thus the uncertainty U SN is codes, including RANS, Navier-Stokes, Euler, boundary-element

estimated. From a general perspective, if we consider the three methods, and others. However, clearly much more work is needed

variables U V , 兩E兩, and U reqd there are six combinations 共assuming for other CFD codes 共such as large-eddy simulations兲, additional

none of the three variables are equal兲: error sources, and alternative error and uncertainty estimation

methods, e.g., single-grid methods and both results for additional

1. 兩 E 兩 ⬍U V ⬍U reqd analytical benchmarks 共especially for nonlinear equations and us-

ing stretched grids兲 for improved definitions of correction factors

2. 兩 E 兩 ⬍U reqd ⬍U V and estimates of orders of accuracy, and alternative strategies to

account for the effects of higher-order terms in RE. Improved

3. U reqd ⬍ 兩 E 兩 ⬍U V

verification procedures once available can be easily incorporated

4. U V ⬍ 兩 E 兩 ⬍U reqd into the present overall verification and validation methodology.

Furthermore, more experience is needed through application for

5. U V ⬍U reqd ⬍ 兩 E 兩 different codes and geometry and conditions, especially for prac-

tical applications.

6. U reqd ⬍U V ⬍ 兩 E 兩 (40) As mentioned in the Introduction, present verification and vali-

dation methodology and procedures were recommended and used

In cases 1, 2, and 3, 兩 E 兩 ⬍U V ; validation is achieved at the U V

at the recent Gothenburg 2000 Workshop on CFD in Ship Hydro-

level; and the comparison error is below the noise level, so at-

dynamics 共Larsson et al. 关11兴兲. 22 participating research groups

tempting to estimate ␦ SM A is not feasible from an uncertainty

from 12 countries and 19 different RANS codes were used for

standpoint. In case 1, validation has been achieved at a level

simulations of 3 test cases representing tanker, container, and sur-

below U reqd , so validation is successful from a programmatic

face combatant hull forms. Most groups implemented the recom-

standpoint.

mended procedures, but lack of familiarity with the procedures

In cases 4, 5, and 6, U V ⬍ 兩 E 兩 , so the comparison error is above

and use of coarse grids led to difficulties. Coarser grid solutions

the noise level and using the sign and magnitude of E to estimate

are far from the asymptotic range and show variability such that

␦ SM A is feasible from an uncertainty standpoint. If U V Ⰶ 兩 E 兩 , then

not all variables display monotonic convergence and oscillatory

E corresponds to ␦ SM A and the error from the modeling assump-

convergence and even divergence is evident. For monotonic con-

tions can be determined unambiguously. In case 4, validation is

vergence, variability in the estimated order of accuracy was ob-

successful at the 兩E兩 level from a programmatic standpoint.

served for some cases. The current 1 million point grids are

Now consider the approach in which the simulation numerical

clearly insufficient for more complex hull forms such as the tanker

error is taken to be deterministic and thus ␦ SN * and the uncertainty

and an order of magnitude increase in points may be required to

U V C are estimated. A similar set of comparisons as those in Eq. remove variability and achieve monotonic convergence for most

共40兲 can be constructed using 兩 E C 兩 , U V C , and U reqd . Since E C variables. In spite of difficulties, the effort was beneficial in en-

can be larger or smaller than E, but U V C should always be less abling quantitative evaluation of levels of verification and valida-

than U V , the results for a given corrected case are not necessarily tion, increasing familiarity with verification and validation proce-

analogous to those for the corresponding uncorrected case. That dures, interpretation of results, and identification of grid

is, a variable can be validated in the corrected but not in the requirements for decreasing levels of errors and uncertainties.

uncorrected case, or vice versa. For cases 4, 5, and 6 in which Careful examination of verification results even for relatively

U V C ⬍ 兩 E C 兩 , one can argue that E C is a better indicator of ␦ SM A coarse grid solutions provides a road map towards achieving ac-

ceptable levels of verification.

than is E, assuming that one’s confidence in using the estimate

Verification and validation methodology and procedures should

␦ SN

* is not misplaced.

be helpful in guiding future developments in CFD through docu-

mentation, verification, and validation studies and in transition of

4.2. Use of Corrected Versus Uncorrected Simulation Re- CFD codes to design through establishment of credibility. Pre-

sults. As previously stated in Section 3.3, the requirements for sumably, with a sufficient number of documented, verified, and

correcting the solution are that the correction factor be close to validated solutions along with selected verification studies, a CFD

one and that confidence in solutions exist. Since the variability of code can be accredited for a certain range of applications. The

the order of accuracy cannot be determined from solutions on contribution of the present work is in providing methodology and

three grids, confidence is difficult to establish in this case. As a procedures for the former, which hopefully will help lead to the

result, caution should be exercised when correcting solutions us- latter.

ing information from only three grids. Part 2 provides an example for RANS simulations for a cargo/

If a validation using the corrected approach is successful at a container ship where issues with regard to practical application of

set condition, then if one chooses to associate that validation un- the methodology and procedures and interpretation of verification

certainty level with the simulation’s prediction at a neighboring and validation results are discussed 共Wilson et al. 关12兴兲.

condition that prediction must also be corrected. That means

enough runs are required at the new condition to allow estimation

of the numerical errors and uncertainties. If this is not done, then Acknowledgments

the comparison error E and validation uncertainty U V correspond- This research was sponsored by the Office of Naval Research

ing to the use of the uncorrected S and its associated 共larger兲 U SN under Grants N00014-96-1-0018, N00014-97-1-0014, and

should be the ones considered in the validation with which one N00014-97-1-0151 under the administration of Dr. E. P. Rood.

wants to associate the prediction at a new condition. 共Whether to The authors gratefully acknowledge Dr. Rood and other col-

and how to associate an uncertainty level at a validated condition leagues, especially Prof. W. G. Steele and Dr. H. Raven, who

with a prediction at a neighboring condition is very much unre- made significant contributions through insightful discussions and

solved and is justifiably the subject of much debate at this time.兲 comments on early drafts. The recent Masters and Ph.D. theses of

Downloaded 12 Jan 2011 to 59.162.23.4. Redistribution subject to ASME license or copyright; see http://www.asme.org/terms/Terms_Use.cfm

Dr. B. Chen, Mr. G. Dolphin and Dr. S. H. Rhee, all at The

University of Iowa, Department of Mechanical Engineering, were

helpful both in the development and testing of the present verifi-

cation and validation methodology and procedures.

Nomenclature

Ck ⫽ correction factor

D ⫽ benchmark data

E,E C ⫽ comparison error, corrected

pk ⫽ order of accuracy

Rk ⫽ parameter refinement ratio

S,S C ⫽ simulation result, corrected

T ⫽ truth

U ⫽ uncertainty estimate

UD ⫽ data uncertainty

U E ,U E C ⫽ comparison error uncertainty, corrected

UI ⫽ iteration uncertainty

Fig. 2 Verification results for first-order numerical solution of

U P ,U P C ⫽ parameter uncertainty 共e.g., grid size G and 1D wave equation. „a… Comparison of true error A À S to esti-

time step T兲, corrected mates from RE, „b… correction factor, and „c… comparison of

U reqd ⫽ programmatic validation requirement 円 A À S C 円 and U SCN , and „d… comparison of 円 A À S 円 and U SN .

U S ,U S C ⫽ simulation uncertainty, corrected

U SM ⫽ simulation modeling uncertainty

U SM A ⫽ simulation modeling assumption uncertainty

U S PD ⫽ simulation uncertainty due to use of previous

data tion; and 共ii兲 a second-order implicit method with second-order

U STE ,U S C TE ⫽ simulation total estimated numerical uncertainty central spatial discretization. Since trends from both schemes are

U SN ,U S C N ⫽ simulation numerical uncertainty, corrected similar, only the results from the first-order scheme are presented.

A combined grid size and time step study was performed where

U V ,U V C ⫽ validation uncertainty, corrected ten solutions were obtained by successively doubling both the grid

⌬x k ⫽ increment in kth input parameter 共e.g., grid size and time step such that ⌬t/⌬x⫽0.5 for all solutions. With this

G and time step T兲 approach, solutions changes are used to estimate total 共temporal

␦

⫽ error

and spatial兲 simulation errors and uncertainties. Accordingly, the

␦* ⫽ error estimate with sign and magnitude generic subscript ‘k’ appearing in expressions for errors and un-

␦ I , ␦ I*

⫽ iteration error, estimate certainties in Section 3.3 is replaced with ‘SN’ in this section

␦ P , ␦ *P⫽ parameter error, estimate where appropriate.

␦ S , ␦ SC⫽ simulation error, corrected

Errors, Uncertainties, and Correction Factors. The con-

␦ SN ⫽ simulation numerical error

␦ SM A ⫽ simulation modeling assumption error cept of a multiplication correction factor was introduced in Sec-

⫽ solution change tion 3.3. The correction factor C k was used to define the numerical

SN ⫽ uncertainty in Eq. 共33兲 or when conditions permit to improve error

error in ␦ *

estimates in Eq. 共32兲 and to define the uncertainty in that error

estimate in Eq. 共34兲. Error and uncertainty estimates given by Eqs.

Appendix A. Analytical Benchmarks 共32兲–共34兲 are tested by numerical solution of analytical bench-

The use of analytical benchmarks for development of the con- marks as well as development of expressions for correction factor.

cept of correction factors as discussed in Section 3.3 is presented Figure 2共a兲 compares the true simulation error E to the three-

in this Appendix. For analytical benchmarks, the modeling error is grid error estimate ␦ RE

* from Eq. 共29兲 versus step size at one

1

zero such that the simulation error is solely due to numerical error. spatial location 共x⫽1 since maximums of numerical error occur

Results are obtained for two analytical benchmarks one- there兲. The three-grid estimate accurately estimates the true error

dimensional 共1D兲 wave and two-dimensional 共2D兲 Laplace equa- E for smaller step sizes, but over predicts E for larger step sizes.

tions. The results for the 2D Laplace equation were qualitatively Closer examination reveals that Eq. 共29兲 over estimates the error

similar to those for the 1D wave equation, which are presented. because Eq. 共30兲 under estimates the order of accuracy, as also

Exact solutions from analytical benchmarks are used to determine shown in Fig. 2共a兲.

the exact simulation numerical error which is compared to esti-

Two definitions for C k were investigated. The first is based on

mates from RE, including use of correction factors. More details

are provided in Stern et al. 关13兴, including single grid error solving equation 共32兲 for C k with ␦ RE* defined in Eq. 共29兲 but

k 1

estimates. replacing p k with the improved estimate p k est , which is provided

Verification of Analytical Benchmarks. For verification us- by Eq. 共35兲 where p k est is an estimate of the limiting order of

ing an analytical benchmark, the simulation error and uncertainty accuracy of the first term of the error expansion equation 共27兲.

are given by ␦ S ⫽S⫺A⫽ ␦ SN and U 2S ⫽U SN 2

, while the corrected Similarly, the second definition of correction factor is based on

simulation error and uncertainty are given by ␦ S C ⫽S C ⫺A⫽ SN estimating ␦ RE

* using the first two terms of the powers series and

k1

and U S2 C ⫽U S2 C N . Simulations are verified if 兩 E 兩 ⫽ 兩 A⫺S 兩 ⬍U SN replacing p k and q k with improved estimates p k est and q k est , which

and corrected simulations are verified if 兩 E C 兩 ⫽ 兩 A⫺S C 兩 ⬍U S C N . is provided by Eq. 共36兲 where p k est and q k est are estimates for

The first-order, linear 1D wave equation models the behavior of limiting orders of accuracy of the first and second terms of the

a more complicated 共nonlinear兲 partial differential equation. The error expansion equation 共27兲 as spacing size goes to zero and the

initial condition is prescribed by a Gaussian function centered at asymptotic range is reached. With this definition, correction fac-

x⫽0.0. Two discretization techniques were studied: 共i兲 first-order tors approach one in the limit of zero spacing size. The estimated

共Euler兲 explicit method with first-order upwind spatial discretiza- values p k est and q k est can be based either on the assumed theoret-

Downloaded 12 Jan 2011 to 59.162.23.4. Redistribution subject to ASME license or copyright; see http://www.asme.org/terms/Terms_Use.cfm

ical order of accuracy p k th and q k th or solutions for simplified Laplace equation analytical benchmark. Their results are consis-

geometry and conditions. In either case, preferably including the tent with our own in showing that uncertainty estimates using Eq.

effects of grid stretching. 共33兲 always bounded the true error. Unlike our own results, their

Figure 2共a兲 also compares the true error E to 共i兲 an uncorrected results indicate that the uncertainty estimate from Eq. 共34兲 failed

three-grid error estimate using Eq. 共29兲 and 共ii兲 corrected esti- to bound the difference in the truth and numerical benchmark for

mates based on Eq. 共32兲 with correction factor defined by Eq. 共35兲 some grid triplets when the apparent order of accuracy was esti-

or 共36兲. Both estimates are closer to E than the uncorrected three mated to be larger than the theoretical value.

grid estimate ␦ RE

* , but for coarser grids C (k1 ) is somewhat too

1

small and C (k2 ) is slightly too large. Figure 2共b兲 shows the same References

trends, but directly compares the exact correction factor E/ ␦ RE * to 关1兴 Freitas, C. J., 1993, ‘‘Editorial Policy Statement on the Control of Numerical

Eqs. 共35兲 and 共36兲. In this case, C k ⬍1 indicates that the leading- Accuracy,’’ ASME J. Fluids Eng., 115, pp. 339–340.

关2兴 AIAA, 1998, Guide for the Verification and Validation of Computational Fluid

order term over predicts 共higher-order terms net negative兲 the er- Dynamics Simulations, G-077-1998.

ror. However, for the general case, C k is equally likely to be ⬍1 or 关3兴 Roache, P. J., 1998, Verification and Validation in Computational Science and

⬎1 depending whether the order of accuracy is approached from Engineering, Hermosa Publishers, Albuquerque, New Mexico.

below or above, respectively. C k ⬎1 indicates that the leading- 关4兴 Mehta, U. B., 1998, ‘‘Credible Computational Fluids Dynamics Simulations,’’

AIAA J., 36, pp. 665– 667.

order term under predicts 共higher-order terms net positive兲 the 关5兴 Stern, F., Paterson, E. G., and Tahara, Y., 1996, ‘‘CFDSHIP-IOWA: Computa-

error. Thus, for the general case the correction to the leading-term tional Fluid Dynamics Method for Surface-Ship Boundary Layers and Wakes

error estimate is equally likely to be positive or negative and can and Wave Fields,’’ Iowa Institute of Hydraulic Research, The University of

be used to define the simulation numerical uncertainty. Iowa, IIHR Report No. 381.

关6兴 Coleman, H. W., and Stern, F., 1997, ‘‘Uncertainties in CFD Code Validation,’’

For C k sufficiently close to 1 and having confidence, ␦ * k and ASME J. Fluids Eng., 119, pp. 795– 803. 共Also see ‘‘Authors’ Closure,’’

U k C are estimated. Correction factors C (k1 ) and C (k2 ) are used to ASME J. Fluids Eng., Vol. 120, Sept. 1998, pp. 635– 636.兲

关7兴 Coleman, H. W., and Steele, W. G., 1999, Experimentation and Uncertainty

estimate the error ␦ * k in Eq. 共32兲 which can then also be used in Analysis for Engineers, 2nd Edition, Wiley, New York, NY.

the calculation of S C 关in Eq. 共10兲兴 and uncertainty U k C in Eq. 共34兲. 关8兴 Rood, E. P., 1996, ‘‘Validation Strategy for RANS Computational Ship Hydro-

dynamics,’’ 2nd International Conference on Hydrodynamics, Hong Kong.

Figure 2共c兲 shows a comparison of 兩 A⫺S C 兩 and three uncertainty 关9兴 ITTC, 1996, 21st ITTC Proceedings, ‘‘Report of the Resistance Committee,’’

estimates: 共i兲 U S( 1C)N defined using C (k1 ) ; 共ii兲 U S( 2C)N defined using Bergen/Trondheim, Norway, September.

关10兴 ITTC, 1999, 22nd ITTC Proceedings, ‘‘Report of the Resistance Committee,’’

C (k2 ) ; and 共iii兲 U S C N from a factor of safety approach given by Eq. Seoul, Korea/Shanghai, China, Sept.

共38兲 with F S ⫽1.25. The results show that the uncertainty estimate 关11兴 Larsson, L., Stern, F., Bertram, V., 2000, ‘‘Gothenburg 2000 A Workshop on

Numerical Ship Hydrodynamics,’’ Chalmers University of Technology, Goth-

U S( 1C)N successfully bounds 兩 A⫺S C 兩 over the entire range of step enburg, Sweden, Sept.

关12兴 Wilson, R. V., Stern, F., Coleman, H., and Paterson, E., 2001, ‘‘Comprehensive

sizes and that U S( 2C)N is not conservative enough since U S( 2C)N ⬍ 兩 A Approach to Verification and Validation of CFD Simulations—Part 2: Appli-

⫺S C 兩 . The uncertainty estimate based on the factor of safety ap- cations for RANS Simulation of A Cargo Container Ship,’’ ASME J. Fluids

proach is not conservative enough for the coarsest two grids and is Eng., 123, published in the issue, pp. 803– 810.

关13兴 Stern, F., Wilson, R. V., Coleman, H., and Paterson, E., 1999, ‘‘Verification and

overly conservative by an order of magnitude for the four finest Validation of CFD Simulations,’’ Iowa Institute of Hydraulic Research, The

grids 共i.e., when solutions are in the asymptotic range兲. For C k University of Iowa, IIHR Report No. 407.

sufficiently less than or greater than 1 and lacking confidence, U k 关14兴 Oberkampf, W. L. and Trucano, T. G., 2000, ‘‘Validation Methodology in

is estimated, but not ␦ * (1)

k and U k C . Correction factors C k and C k

(2) Computational Fluid Dynamics,’’ AIAA Fluids 2000, Paper No. 2549, Denver,

CO.

are used to estimate the uncertainty in Eq. 共33兲 which is compared 关15兴 Coleman, H. W., Stern, F., Mascio, A. Di, and Campana, E., 2001, ‘‘The

to factor of safety approach given by Eq. 共37兲. Figure 2共d兲 shows Problem with Oscillatory Behavior in Grid Convergence Studies,’’ ASME J.

Fluids Eng., 123, No. 2, pp. 438 – 439.

that all three uncertainty estimates successfully bound the true 关16兴 Hoekstra, M., Eca, L., Windt, J., and Raven, H., 2000 ‘‘Viscous Flow Calcu-

error 兩 A⫺S 兩 although the factor of safety approach is overly con- lations for KVLCC2 AND KCS Models Using the PARNASSOS Code,’’ Pro-

servative for all grids. ceedings Gothenburg 2000 A Workshop on Numerical Ship Hydrodynamics,

Uncertainty estimates enable a quantitative measure of verifica- Gothenburg, Sweden.

关17兴 Ferziger, J. H., and Peric, M., 1996, Computational Methods for Fluid Dynam-

tion for analytical benchmarks. Figure 2共c, d兲 indicates that the ics, Springer-Verlag, New York.

present solutions are verified over the chosen range of grid size 关18兴 Eca, L., and Hoekstra, M., 2000, ‘‘On the Application of Verification Proce-

and time step. As expected, the largest levels of uncertainty are for dures in Computational Fluid Dynamics,’’ 2nd MARNET Workshop.

the coarsest grid size and time step where levels are (U k ,U k C ) 关19兴 Ebert, M. P., and Gorski, J. J., 2001, ‘‘A Verification and Validation Procedure

for Computational Fluid Dynamics Solutions,’’ NSWCCD-50-TR-2001/0006,

⫽(15%,7.5%). Hydromechanics Directorate Report, NSWC, Carderock Division, West Be-

Eca and Hoekstra 关18兴 also perform verification for the 2D thesda MD 20817-5700.

Downloaded 12 Jan 2011 to 59.162.23.4. Redistribution subject to ASME license or copyright; see http://www.asme.org/terms/Terms_Use.cfm

- 455984.fucak_fpUploaded byZoric Bobby
- FEA Basics very imp.pptUploaded bysandeshlikes
- Rich Text Editor FileUploaded byI'm a moron.
- CFD hvacUploaded byGopinath Gopi
- M_Cely_0529Uploaded byMarcela Cely Santos
- Seakeeping prediction of KVLCC2 in head waves with RANS.pdfUploaded byWai-Yen Chan
- NAPA Product2008Uploaded bycornelgg474469
- p19 Cala Unce PoUploaded byedwardlist
- Workshop Watermatex CFDUploaded byLuay Kadhum Alwaili
- Uncertainty Budget Table Template 16Jan2013Uploaded byBALAJI
- Local_Scour_Free_Surface_Mesh.pdfUploaded byjdj2007
- CFD Computational Modeling of Nearfield to Farfield Plume Expansion AIAA-2007-5704.pdfUploaded bymojicap
- syllabus cfd.pdfUploaded byAttique Javaid
- Workshop_watermatex_CFD.pdfUploaded byLuay Kadhum Alwaili
- Sim MultiphysicsUploaded byLUISALBERTO06011985
- SGC2016 NortonStraw MattStraw PublicUploaded bypmr80_2004
- Process IntensifierUploaded byEric Toro
- 3D-Free-Surface-Wave-Analysis-CFD-7.pdfUploaded bybrennomourabmc
- hgODIJAIOF2BUploaded byKarthik Abhi
- Uso Su Tre Casi Dei Modelli Cfd Per Prevedere l'Andamento Dei Fumi in Ambienti Chiusi ComplessiUploaded byAndrea Nicola Turcato
- Synopsis Presentation on HPHE9.45 16.02.18Uploaded byyogesh shinde
- 252 CP Rubric GuideUploaded byJayParmar
- ScienceUploaded byRodrigo Gonçalves
- Bochum Comp Mech SubjectUploaded byrohan
- Variable Intake & Exhaust PresentationUploaded byOmar Nasser Khalaf
- GeorgeUploaded byPurushothaman
- 1757-482x.2.1.33.pdfUploaded byalex
- 2016 Article A Study Of Airfoil ParameterizationUploaded byPablo Chedas
- Physics EnvyUploaded bymissclarej
- Physics Envy Andrew LoUploaded byAdam Gouveia

- ANSYS Turbomachinery CFD System - 14.5 UpdateUploaded byPrabhuMurugesan
- CascadeUploaded bymistrycs
- Fluid 12Uploaded byfadapow4u
- Mother's Voice #1Uploaded bymistrycs
- Inlets for High Angle s of AttackUploaded bymistrycs
- Averaging Pitot Tubes; Fact and FictionUploaded bymistrycs
- tpgs.pdfUploaded bySudipta Dubey
- energies-07-08150Uploaded bymistrycs
- Centrifugal-Compressor1 R-H-aungier.pdfUploaded bymistrycs
- Outdoor GamesUploaded bymistrycs
- Full Page PhotoUploaded bymistrycs
- blade profile.pdfUploaded bymistrycs
- 1.pdfUploaded bymistrycs
- Full Page PhotoUploaded bymistrycs
- Turboexpo-2017Uploaded bymistrycs
- Blade ProfileUploaded bymistrycs
- Bladegen-ToturialsUploaded bymistrycs
- Advance Gas Turbine TheoryUploaded bymistrycs
- Leehamnews.com-Bjorns Corner Turbofan Engine Challenges Part 7-2Uploaded bymistrycs
- Spakovsky Waitz. Thermodynamics and PropulsionUploaded bymistrycs
- Spakovsky Waitz. Thermodynamics and PropulsionUploaded bymistrycs
- FACTORS WHICH INFLUENCE THE ANALYSIS AND DESIGN OF EJECTOR NOZZLESUploaded bymistrycs
- Pressure MeasurementsUploaded bymistrycs
- nozzle designUploaded byramkumar121
- Analytical and Experimental Studies of ASME Flow NozzlesUploaded bymistrycs

- NumerologyUploaded byscribd_new_account
- INTRO_Labsguru_LGRB003Uploaded bykanak kalita
- Multi-Tenancy_and_Hosting_Guidance_For_Exchange_Server_2010_SP2.docxUploaded byDev San
- North American Literature PoemsUploaded by여자마비
- David Wallace, Quantum Probability From Subjective LikelihoodUploaded byCarlosGerardoSanjuánCiepielewski
- Cheg FundamentalsUploaded bytesfay gebre
- Iap Guide Book on Immunization Immunization in Special SituationsUploaded byNugrah Tri Amiranti
- Karate and Oriental Arts 65 magazineUploaded bypaul miller
- declarationofindepencecommonlitUploaded byapi-262173560
- A. Drug StudyUploaded byyfc22
- critical analysis-kwicholUploaded byapi-287050484
- 1 CorinthiansUploaded byDavid Philip Alinas
- Factoring is un-multiplying, so in order toUploaded bystokesr5257
- ECE 1201 Semiconductors and DiodesUploaded byA.K.M.TOUHIDUR RAHMAN
- [Justin Raden] the Anxiety of the Contemporary (2017)Uploaded bymariuszabo
- ch 8 sect 2 solve by addition 014Uploaded byapi-314809600
- PHP Login System With Admin FeaturesUploaded bysnkhan120
- Placebo EffectUploaded bysmarter
- Derech Hateva 18Uploaded byYeshiva University
- A comprehensive analysis of biodiesel impact on exhaust emissionUploaded byEnvBiotech
- Sandeep Interim Thesis ReportUploaded bySandeep Bethala
- Assembly raisesUploaded byRick Karlin
- An Enhanced Routing Method With Dijkstra Algorithm and AHP Analysis in GIS-Based Emergency PlanUploaded byBharadwaj K Bardi
- (Exce) Pipe Flow 2-Multiphase Flow Assurance-Ove Bratland-2010Uploaded byHoang Nguyen Minh
- ias32Uploaded byLeonardo Brito
- Development of a Conducted EMI Model for a Industrial PowerUploaded byBrehaut
- The Power of Touch in Media by PHD UKUploaded byanjali_ramachan2848
- Manotoc v CA.pdfUploaded byIcon Montius
- (Illustrated Classics) Herman Melville-Moby Dick-Saddleback Educational Publishing (2007)Uploaded byalex_xNa
- sst assessment reportUploaded byapi-241979067