14 views

Uploaded by Vinayak Gururaj

- ISO 9001:2008 Edition
- 'The essential link between ethnographic research, human factors engineering and risk management in injection device development' by Alun Wilcox.
- Software Testing for Medical Devices
- Null Hypothesis, Statistics, Z-Test, Significance
- Statistical Significance Hypothesis
- Testing for Segregation Distortion in Genetic Scoring-NATURE
- Software Testing - 2
- Brm Unit 1 New 2007
- BRM unit1
- tmp8752.tmp
- 1-s2.0-S0309170805002903-main.pdf
- 79e4150bf71d4edcd1
- 37-49-1-PB.pdf
- The Effect of Intellectual Capital Performance to Company's Financial Performance: An Empirical Study in Financial Sector Non-Banking Companies Listed in IDX
- Modelling and Simulation Complete
- Highlights of ISO 9001 2008 Changes Subodh Hegde
- ch02 (2)
- example+of+journal+critique
- 2-Handout-ISO-9001-Clause-Map-2015-to-2008
- PTCG_HandS_April0808.pdf

You are on page 1of 36

of Simulation Models

Dr. A. K. Dey

1

Verification and Validation

of Simulation Models

right. It is utilized in the comparison of the

conceptual model to the computer representation

that implements that conception.

It asks the questions: Is the model implemented

correctly in the computer? Are the input

parameters and logical structure of the model

correctly represented?

2

Verification and Validation

of Simulation Models

model. It is utilized to determine that a model is

an accurate representation of the real system.

Validation is usually achieved through the

calibration of the model, an iterative process of

comparing the model to actual system behavior

and using the discrepancies between the two, and

the insights gained, to improve the model. This

process is repeated until model accuracy is judged

to be acceptable.

3

Model building, verification & validation

First step

Observe the real system

Observe interactions among their various components

Collect data on their behaviour

Second step

Construct a conceptual model: a collection of assumptions

about the components and the structure of the system, plus

hypotheses about the values of the model input parameters

It is comparison of the of the real system to the conceptual

model

Third step

Implementation of a operational model by using a

simulation software

4

Verification of Simulation

Models

use in the verification process.

1. Have the code checked by someone other than

the programmer.

2. Make a flow diagram which includes each

logically possible action a system can take when

an event occurs, and follow the model logic for

each action for each event type.

5

Verification of Simulation

Models

reasonableness under a variety of settings of the

input parameters. Have the code print out a wide

variety of output statistics.

4. Have the computerized model print the input

parameters at the end of the simulation, to be

sure that these parameter values have not been

changed inadvertently.

6

Verification of Simulation

Models

possible. Give a precise definition of every

variable used, and a general description of the

purpose of each major section of code.

any programmer would follow when debugging a

computer program.

7

Calibration and Validation

of Models

C o m p a re m o d e l

In itia l

to re a lity M odel

R e v is e

C o m p a re

re v is e d m o d e l

F irs t re v is io n

R eal of m odel

to re a lity

S y s te m R e v is e

C o m p a re 2 n d

re v is e d m o d e l Second

re v is io n

to re a lity of m odel

R e v is e

<Iterative process of calibrating a model>

8

Validation of Simulation Models

As an aid in the validation process, Naylor and Finger

formulated a three-step approach which has been

widely followed:

1. Build a model that has high face validity

Should appear to be reasonable to model users and other

knowledgeable (about the real system) people

Users may be asked if the model behaves in the expected

manner if the input values are changed (sensitivity analysis)

If there are many input variables, select critical variables for

tests

9

Validation of Simulation Models

2. Validate model assumptions: Two classes of assumptions- Structural

and Data

Structural:

Number of lines, One line for each server or one line fro multiple

servers, Customers could change lines if one line is moving faster, No.

of tellers could be fixed or variable

These should be verified by actual observations at appropriate time

periods also by knowing the policies of the organization

Data

Collection of reliable data and correct statistical analysis

Examples: Inter arrival time during peak and slack periods, service

times for corporate and personal accounts

Homogeneity tests and correlation tests should be carried out

Homogeneity: Do data sets collected at two different peak hours have come from

the same population? If so, the two sets can be combined

Correlation: As soon as the analyst is assured of dealing with a random sample (i.e.,

correlation is not present) the statistical analysis can begin

10

Validation of Simulation Models

Identify an appropriate probability distribution

Estimate the parameters of the hypothesized distribution

Validate the assumed statistical model by a goodness-of-fit test: Chi Square or

K – S tests and by graphical method

Use of goodness-of-fit test is an important part of validation of

assumptions

3. Compare the model input-output transformations to

corresponding input-output transformations for the real system.

11

Validation of

Model Assumptions

Structural

involves questions of how the system operate

(Example1)

Data assumptions should be based on the collection of reliable

data and correct statistical analysis of the data.

Customers queueing and service facility in a bank (one line or

many lines)

1. Interarrival times of customers during several 2-hour

periods of peak loading (“rush-hour” traffic)

2. Interarrival times during a slack period

3. Service times for commercial accounts

4. Service times for personal accounts

12

Validation of

Model Assumptions

The analysis of input data from a random sample

consists of three steps:

1. Identifying the appropriate probability distribution

2. Estimating the parameters of the hypothesized

distribution

3. Validating the assumed statistical model by a

goodness-of fit test, such as the chi-square or

Kolmogorov-Smirnov test, and by graphical

methods.

13

Validating Input-Output

Transformation

(Example) : The Fifth National Bank of Jaspar

The Fifth National Bank of Jaspar, as shown in the

next slide, is planning to expand its drive-in

service at the corner of Main Street. Currently,

there is one drive-in window serviced by one teller.

Only one or two transactions are allowed at the

drive-in window, so, it was assumed that each

service time was a random sample from some

underlying population. Service times {Si, i = 1,

2, ... 90} and interarrival times {Ai, i = 1, 2, ... 90}

14

Validating Input-Output

Transformation

Y 'a ll C o m e

B ack Bank

M a in S tr e e t

STOP

Jaspar

Main Street

Drive-in window at the W e lc o m e

Fifth National Bank. to J a s p a r

15

Validating Input-Output

Transformation

were collected for the 90 customers who arrived

between 11:00 A.M. and 1:00 P.M. on a Friday.

This time slot was selected for data collection

after consultation with management and the teller

because it was felt to be representative of a

typical rush hour. Data analysis led to the

conclusion that the arrival process could be

modeled as a Poisson process with an arrival

rate of 45 customers per hour; and that service

times were approximately normally distributed

with mean 1.1 minutes and

16

Validating Input-Output

Transformation

standard deviation 0.2 minute. Thus, the model

has two input variables:

1. Interarrival times, exponentially

distributed (i.e. a Poisson arrival

process) at rate λ = 45 per hour.

2. Service times, assumed to be N(1.1,

(0.2)2)

17

Validating Input-Output

Transformation

Poisson arrivals Teller’s utilization

rate = 45/hour

X11 , X12,... M Y1 =

Random

variables

O

Service times

X21 , X22,... D Average delay

N(D2,0.22)

E Y2

One teller L

D1 = 1

Decision Mean service time “black Maximum line length

variables box”

D2 = 1.1 minutes Y3

One line

D3 = 1

Input variables Model Output variables

Validating Input-Output

Transformation

The uncontrollable input variables are

denoted by X, the decision variables by D,

and the output variables by Y. From the

“black box” point of view, the model takes

the inputs X and D and produces the

outputs Y, namely

(X, D) f Y

or

f(X, D) = Y

19

Validating Input-Output

Transformation

Input Variables Model Output Variables, Y

X = other variables to management (Y1, Y2, Y3)

Poisson arrivals at rate Y1 = teller’s utilization

= 45 / hour Y2 = average delay

X11, X12,.... Y3 = maximum line length

Input and Output variables for model of current bank operation (1)

20

Validating Input-Output

Transformation

Input Variables Model Output Variables, Y

X21 , X22 ,..... secondary interest

Y4 = observed arrival rate

D1 = 1 (one teller) Y5 = average service time

D2 = 1.1 min Y6 = sample standard deviation

(mean service time) of service times

D3 = 1 (one line) Y7 = average length of line

Input and Output variables for model of current bank operation (2)

21

Validating Input-Output

Transformation

Statistical Terminology Modeling Terminology Associated Risk

when H0 is true

Type II : failure to reject H0 Failure to reject an β

when H0 is false invalid model

and vice versa, given a fixed sample size.

Once α is set, the only way to decrease β is to increase

the sample size.

22

Validating Input-Output

Transformation

Y4 Y5 Y2 =avg delay

Replication (Arrivals/Hours) (Minutes) (Minutes)

1 51 1.07 2.79

2 40 1.12 1.12

3 45.5 1.06 2.24

4 50.5 1.10 3.45

5 53 1.09 3.13

6 49 1.07 2.38

sample mean 2.51

standard deviation 0.82

23

Validating Input-Output

Transformation

Z2 = 4.3 minutes, the model responses, Y2. Formally,

a statistical test of the null hypothesis

H0 : E(Y2) = 4.3 minutes

versus ----- (Eq 1)

H1 : E(Y2) ≠ 4.3 minutes

is conducted. If H0 is not rejected, then on the

basis of this test there is no reason to consider

the model invalid. If H0 is rejected, the current

version of the model is rejected and the modeler

is forced to seek ways to improve the model, as

illustrated in Table 3.

24

Validating Input-Output

Transformation

As formulated here, the appropriate statistical test is

the t test, which is conducted in the following

manner:

Step 1. Choose a level of significance α and a

sample size n. For the bank model, choose

α = 0.05, n = 6

Step 2. Compute the sample mean Y2 and the

sample standard deviation

n S over the n replications.

Y2 = {1/n} i=Σ1Y2i = 2.51 minutes

25

Validating Input-Output

Transformation

and

n

S = { i=Σ1 (Y2i - Y2)2 / (n - 1)}1/2 = 0.82 minute

where Y2i , i = 1, .., 6, are shown in Table 2.

Step 3. Get the critical value of t from Table A.4. For

a two-sided test such as that in Equation 1, use

tα /2,n-1 ; for a one-sided test, use tα ,n-1 or -tα ,n-1 as

appropriate (n -1 is the degrees of freedom). From

Table A.4, t0.025,5 = 2.571 for a two-sided test. 26

Validating Input-Output

Transformation

Step 4. Compute the test statistic

t0 = (Y2 - µ 0) / {S / √n} ----- (Eq 2)

where µ 0 is the specified value in the null

hypothesis, H0 . Here µ 0 = 4.3 minutes, so that

t0 = (2.51 - 4.3) / {0.82 / √6} = - 5.34

Step 5. For the two-sided test, if |t0| > tα /2, n-1 , reject H0

. Otherwise, do not reject H0. [For the one-sided

test with H1: E(Y2) > µ 0, reject H0 if t > tα , n-1 ; with

H1 : E(Y2) < µ 0 , reject H0 if t < -tα , n-1 ]

27

Validating Input-Output

Transformation

Since | t | = 5.34 > t0.025,5 =2.571, reject H0 and

conclude that the model is inadequate in its

prediction of average customer delay.

the null hypothesis H0 is a strong conclusion,

because P(H0 rejected | H0 is true) = α

28

Validating Input-Output

Transformation

Y4 Y5 Y2 =avg delay

Replication (Arrivals/Hours) (Minutes) (Minutes)

1 51 1.07 5.37

2 40 1.11 1.98

3 45.5 1.06 5.29

4 50.5 1.09 3.82

5 53 1.08 6.74

6 49 1.08 5.49

sample mean 4.78

standard deviation 1.66

29

Validating Input-Output

Transformation

Step 1. Choose α = 0.05 and n = 6 (sample size).

Step 2. Compute Y2 = 4.78 minutes, S = 1.66

minutes ----> (from Table 3)

Step 3. From Table A.4, the critical value is

t0.025,5 = 2.571.

Step 4. Compute the test statistic

t0 = (Y2 - µ 0) / {S / √n} = 0.710.

Step 5. Since | t | < t0.025,5 = 2.571, do not reject H0 , and

thus tentatively accept the model as valid.

30

Validating Input-Output

Transformation

To consider failure to reject H0 as a strong

conclusion, the modeler would want β to be

small. Now, β depends on the sample size n

and on the true difference between E(Y2) and µ 0

= 4.3 minutes, that is, on

δ = | E(Y2) - µ 0 | / σ

where σ , the population standard deviation of

an individual Y2i , is estimated by S. Table A.9 and

A.10 are typical operating characteristic (OC)

curves, which are graphs of the probability of a

31

Validating Input-Output

Transformation

Type II error β (δ ) versus δ for given sample

size n. Table A.9 is for a two-sided t test while

Table A.10 is for a one-sided t test. Suppose that

the modeler would like to reject H0 (model

validity) with probability at least 0.90 if the true

means delay of the model, E(Y2), differed from

the average delay in the system, µ 0 = 4.3

minutes, by 1 minute. Then δ is estimates by

δ = | E(Y2) - µ 0 | / S = 1 / 1.66 =

0.60

32

Validating Input-Output

Transformation

For the two-sided test with α = 0.05, use of Table

A.9 results in

β (δ ) = β (0.6) = 0.75 for n = 6

To guarantee that β (δ ) ≤ 0.10, as was desired

by the modeler, Table A.9 reveals that a sample

size of approximately n = 30 independent

replications would be required. That is, for a

sample size n = 6 and assuming that the

population standard deviation is 1.66, the

probability of accepting H0 (model validity) , when

in fact the model is invalid

33

Validating Input-Output

Transformation

(| E(Y2) - µ 0 | = 1 minute), is β = 0.75, which is

quite high. If a 1-minute difference is critical, and

if the modeler wants to control the risk of

declaring the model valid when model predictions

are as much as 1 minute off, a sample size of n =

30 replications is required to achieve a power of

0.9. If this sample size is too high, either a higher

β risk (lower power), or a larger difference δ ,

must be considered.

34

35

36

- ISO 9001:2008 EditionUploaded byGaurav Narula
- 'The essential link between ethnographic research, human factors engineering and risk management in injection device development' by Alun Wilcox.Uploaded byPDD
- Software Testing for Medical DevicesUploaded byandrewd42
- Null Hypothesis, Statistics, Z-Test, SignificanceUploaded bySherwan R Shal
- Statistical Significance HypothesisUploaded bySherwan R Shal
- Testing for Segregation Distortion in Genetic Scoring-NATUREUploaded byvahidvashar
- Software Testing - 2Uploaded byvinitajenny
- Brm Unit 1 New 2007Uploaded byPreethi Rajasekaran
- BRM unit1Uploaded byShiva Kumar
- tmp8752.tmpUploaded byFrontiers
- 1-s2.0-S0309170805002903-main.pdfUploaded byThefirefox
- 79e4150bf71d4edcd1Uploaded byNikunj Jadawala
- 37-49-1-PB.pdfUploaded byNi'ma Salisa
- The Effect of Intellectual Capital Performance to Company's Financial Performance: An Empirical Study in Financial Sector Non-Banking Companies Listed in IDXUploaded byRendra M. Jenk Brotherz
- Modelling and Simulation CompleteUploaded byJasperFabella
- Highlights of ISO 9001 2008 Changes Subodh HegdeUploaded bySUBODHH
- ch02 (2)Uploaded byb87208582
- example+of+journal+critiqueUploaded byDanialAiman
- 2-Handout-ISO-9001-Clause-Map-2015-to-2008Uploaded byLamtua Purba
- PTCG_HandS_April0808.pdfUploaded byzilangamba_s4535
- 9-100 Intro to the StandardUploaded byvp_s4460
- TypesUploaded byKimgenesis Tuazon
- An Innovative Approach to Investigate VaUploaded byAlvaro Daniel Berrios Zuniga
- chap09_01.pptUploaded byMahmoud Wehd
- Testing)Uploaded bySurendra Singh Chauhan
- BASIC CONCEPTS in testing hypothesis.pptxUploaded byjulie anne macuse
- Hypothesis TestingUploaded byPallavi Shettigar
- CT3_Sol_0416Uploaded byNathan Rk
- From concepts to practice: combining different approaches to understand drivers of landscape changeUploaded byJesús Muñoz Serrano
- FMCG.docxUploaded byKrishnamohan Vaddadi

- Formula NotationUploaded byShekh Aurangzeb
- Backpropagation Neural Network for XOR Problem Java source codeUploaded byDenny Hermawanto
- radiology quality management.docxUploaded byselinasimpson361
- Elmasri_6e_Ch04Uploaded byJoona John
- Dbms Lab MaterialUploaded byThanmay Konda
- OCL-toolsUploaded byTHOMASB_19747048
- Notes on Class 2.pdfUploaded byHasan Hasib
- clientUploaded byGeorshua Melendez
- 3_statistik_cukup.pptUploaded byyulia sari
- Oracle FinalUploaded byDaniel Anghel
- OpenCV Lections: 6. ContoursUploaded byDenis Perevalov
- dip prgUploaded byAmit Shukla
- assignment_Part1_SP22_2014 (1)Uploaded byDeepak Jaiswal
- 480-1 (Manski, Northwestern).pdfUploaded byInvest
- Berger Peric Chi Wiley 3Uploaded bycamilo soto
- Database ModelsUploaded byswati11319
- TensorFlow BasicsUploaded byAvinash
- OODBs vs ORDBsUploaded bythomas.keschl8224
- Free AutoCAD Tutorials _ Isometric Drafting in AutoCAD 2010Uploaded byVivian Kallou
- Econometrics MadalaUploaded byHimadri Palikhe
- DoDAF V2 - Volume 2Uploaded byscranidi
- WMS - 334 Application Developent MethodsUploaded byTangeni D Grote
- SimulationUploaded bySadiaAslam
- Pertemuan 6Uploaded byMuhammad Taufiq
- NormalizationUploaded byMajid Abdullah
- Lesson 1Uploaded byniranjanusms
- UML SlidesUploaded byTuhin Utsab Paul
- Chapter 11Uploaded byVijendhar Reddy
- New Rich Text Document (2)Uploaded byMuhammad Tahir
- Solutions to Problems related to Information theoryUploaded byRohit Bhadauria