8 views

Uploaded by Fausto De La Cadena

- Chapter IV Sri bisa dijaikan bahasan
- Computational Social Science and System Dynamics
- MBA First & Second Year Syllabus - Sub Centre Latur
- Six Sigma
- QNT 273 Academic Success/snaptutorial
- Study on Gis Simulated Water Quality Model
- SimulationModels
- Project Rubric
- qch9-10
- Modeling of Biological Manufacturing Systems for Dynamic Reconfiguration_uremovic
- Sampling Distributions
- 61041
- Biostatistics (Dr Shilpi Gilra)
- Solutions Manual Probability and Statistics for Engineers and Scientists Manual Hayler
- Perverse Nature of SD
- Zipper Merge Evaluation of the Late Merge Traffic Work Zone 05-r6
- Lazear, Rosen ('81) - Rank Order Tournaments
- Conditional Simulations of Fluid Flow in Three-dimensional Networks of Discrete Fractures
- 122665_Lecture 7a Synthesis Phase
- Operation Management.pdf

You are on page 1of 38

Chapter 8

Model Goodness: Verification and Validation

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 1

Verification assesses the correctness of the formal representation of the intended model (in our case, a computer simulation program), by inspecting computer code and test runs, and performing consistency checks on their statistics Validation assesses how realistic the modeling assumptions are, by comparing model performance metrics (predictions), obtained from model test runs, to their counterparts in the system under study

obviously, validation is possible only if the simulation program exists

Verification activities via inspection of test runs include

input parameters and output statistics using a debugger using animation

sanity checks

WIP inventory Processor Job Arrivals Job Departures

A simple workstation

A Generic workstation can be described as a queueing system the flow of incoming jobs, either singly or in batches, forms the arrival stream the time a job is delayed for processing at the workstation server is its service time. if an incoming job cannot start processing right away (because servers are busy),

then it is held in a buffer (finite or infinite), and in due time will be removed from the buffer and assigned a server (real-life workstations it is, of course, finite) a job that finds the buffer full on arrival is usually rerouted to another workstation or simply assumed to be lost the order in which jobs queue up in the buffer is called a queueing discipline. a workstation model may have additional wrinkles, such as server failure and repair, routinely modeled via random uptimes and random downtimes

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 4

The arrival process is denoted by {A } , where A is the i-th interarrival i i interval, separating customers i-1 and i

for stationary arrival processes, the rate is denoted by l = 1 / E [A ]

i

i

i

A standard notation has been developed to specify a queueing system succinctly, employing slash-separated symbols, representing arrival process, service process, number of servers and queue capacity in this order (an omitted capacity is understood to be infinite)

the symbol GI stands for a general iid distribution (special cases are M for an exponential distribution and D for a deterministic distribution) the symbol G stands for a general distribution non- iid distribution M/M/1 specifies a queue with iid exponential distributions for inter-arrival and service times (Poisson process), a single server and infinite buffer capacity M/D/k/C specifies a queue with iid exponential inter-arrival times, deterministic service times, k servers and a system of capacity of C, of which the first k positions are occupied by servers

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 5

Service Disciplines

Common service disciplines:

FIFO (First-In-First-Out), also known as FCFS (First-Come-FirstServed) is the most common discipline, where jobs are served in order of arrival LIFO (Last-in-First-Out), also known as LCFS (Last-Come-FirstServed), serves the most recent arrival first SIRO (Service-In-Random-Order) selects the next job in the buffer randomly, with equal probabilities RR (Round Robin) has associated with it a (fixed) service time, often referred to as the time quantum, and jobs are served cyclically, one quantum at a time, until attaining their requisite service time PS (Priority-Service) assumes that jobs have priorities associated with them, and selects a job with the highest service priority jobs, such as FIFO, LIFO, SIRO, etc.

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 6

Common queueing performance measures:

average number of jobs in the queue (buffer only) average number of jobs in the system (buffer and servers) average job waiting time (buffer only delay) average job sojourn time (buffer and service delays) server utilization (fraction of time a server is busy) throughput (output rate of the system)

Denote the following quantities as follows:

N s is the average number of jobs in the system, including queue (buffer) and servers

N q is the average number of jobs in the queue (buffer) only

For a queueing system in steady state of capacity (buffer size) C (finite or infinite),

N S = n Pn

n=1 C

N q = N s - (1 - P 0 ) where 1 - P 0 is the steady-state average number of jobs in service

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 8

In general, the server utilization, r , of a single-server queueing system with any queue capacity is given by r = 1 - P 0

For infinite capacity systems, the utilization can be expressed as the ratio of the input rate and the service rate, that is, r = l / m

When r 1, then system is said to be unstable, and the server is said to be exhausted

In this case, the server is not able to keep up with the incoming workload, and consequently the number of jobs in the system grows without bound in the long run (system explosion)

stability is needed for long-run performance measures to exist in the context of simulation modeling we are typically interested in stable systems a finite-capacity system is always stable, because the number of jobs in it is bounded by its capacity.

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 9

A regenerative queueing system is one where temporal histories can be decomposed into periods (random intervals), such that the associated stochastic processes are independent over distinct periods

the onset of a busy cycle is called a regeneration point, since the system regenerates itself there probabilistically each independent period is called a regeneration cycle (or busy cycle), and extends over a pair of consecutive busy period (when the server is busy) and idle period (when the server is idle), as shown below

Number In the System Idle Empty-and-Idle Busy T ime

Empty-and-Idle

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 10

Consider a regeneration cycle in a GI/GI/1 queue, and define

T B is the length of the busy period T

C

Since the number of incoming and outgoing jobs in a regeneration cycle must coincide, one has

l E [T ] = m E [T ] or equivalently E [T ]/ E [T ] = l / m = r

C B B C

It follows that Pr{server is busy } = r = l / m For Poisson arrivals (iid exponential interarrival intervals)

Since residual interarrival intervals are exponential too, 1/ l Pr{server is idle } = 1 - r = E [T ] C E [X ] E [A ] 1/ m 1/ l i i = = Thus, E [T B ] = and E [T C ] = 1- r 1- r 1- r 1- r

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 11

The throughput of a queueing system is its long-run output rate (jobs served per unit time), given by

o = m(1 - P 0 )

which is the service rate while the server is busy For an infinite capacity queue, this becomes

o = l

by flow conservation

12

Littles Formula

Littles Formula is a key general conservation equation, linking the mean number of jobs in the system, N , with the arrival rate, l , and the mean waiting time,W , given by

N = l W

note carefully that l is not the offered arrival rate, but the effective arrival rate, excluding any lost jobs! when the capacity is infinite (so no losses are incurred), then the offered arrival rate and effective arrival rate are the same

for the entire queueing system (buffer plus servers), N for the buffer subsystem only, N

b

S

= l W

= lW

13

Define the following quantities:

p is the long-run fraction of jobs that find k jobs in the system on k arrival. Such quantities are referred to as customer-average probabilities arrival point probabilities. Note carefully that the number of jobs in the system is computed only at the (random) times of arrival and excludes the arriving job. For this reason this kind of system state is referred to as the state embedded at arrival times, or as the state seen by arriving jobs.

P is the long-run fraction of time that the system has precisely k k jobs in it. Such quantities are referred to as time-average probabilities or arbitrary time probabilities.

Generally p k Pk

For single-server queues with finite capacity, K ,

l (1 - p K ) = m(1 - P 0 )

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 14

While generally p k Pk , under certain conditions the equality

p = P , k = 0, 1, K ,

does hold

The most common case where the equality above holds is when the arrival stream is a Poisson process

This case is known as PASTA (Poisson Arrivals See Time Averages)

ASTA (Arrivals See Time Averages) It is possible to have ASTA which is not PASTA

15

As an example, consider an Arena model of a single workstation below

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 16

Arena Queues summary statistics report for the single workstation model

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 17

Arena Resources Summary statistics report for the single workstation model

18

Arena User Specified statistics report for the single workstation model

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 19

Single-Station Verification

For a replication r, define

D (r ) is the number of departures (jobs processed) during replication r

o 1 (r ) =

D (r ) T (r )

The verification procedure to be outlined uses various sections of the Arena summary reports to estimate the throughput in multiple alternative computations, to show that they are all in reasonable agreement

note that is just circumstantial evidence of verification any major discrepancy among the computations would suggest an error in the simulation program (bug) and will preclude verification

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 20

Throughput estimation from the Queues report and the Counter section in the User Specified report

an examination of the previous report shows that replication 1 ran for 100,000 hours, and the Counter section reveals that D(1) = 20,736 jobs were processed and 1,503 jobs were rejected. the first estimate of the throughput is

o 1(1) = 20, 736 / 100, 000 = 0.2074 jobs/hour

Throughput estimation from the Resources section and the throughput equation

an examination of the Resources report shows that the utilization was estimated as u = 0.8358 , and while the workstation was busy, it processed jobs at the rate of 0.25 jobs per hour by the throughput equation o = m(1 - P 0 ) , a second estimate of the throughput is o 2 (1) = mu = 0.25 0.8358 = 0.2090 jobs/hour

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

21

Throughput estimation from the Tally section in the User Specified report

using the mean inter-departure time from the Tally section, a third estimate of the throughput is

o 3 (1) = 1 E int er depar t ur e t im e = 1 / 4.8226 = 0.2074 jobs/hour

Throughput estimation from the Counter section in the User Specified report and the effective arrival rate

the Counter section shows that the number of jobs that succeeded in entering the workstation was 20,736 out of a total of 20736+1503 jobs that arrived at the workstation, so that we estimate 1- p = 20736 / ( 20736 + 1503) = 0.9324 K since the throughput is the effective arrival rate, a fourth estimate is just o 4 (1) = l (1 - p K ) = (1/ 4.5) 0.9324 = 0.2072 jobs/hour

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 22

Littles formula provides another common verification check, using the summary statistics in the previous report as follows:

for the buffer subsystem, Littles formula is N = l W

b eff b

= 1. 3265

b

= 6. 3969

from the Counter Section of the User Specified report, the = 0.2074 effective arrival rate (equal to throughput ) is l

eff

eff

b

= 1. 3265 and l

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 23

eff

As an example, consider the Arena model of a network of two workstations (assembly and painting) in tandem, shown below

Assembly Painting

suppose that neither workstation has buffer capacity limitations assume further that the arrival stream consists of two types of jobs: type 1 and type 2, each with its own assembly and painting operations job inter-arrival times are exponentially distributed with means 4 hours for type 1 and 10 hours for type 2. for type 1 jobs, assembly times (in hours) are distributed according to the Tria(1,2,3) distribution, while painting times are deterministic lasting 3 hours for type 2 jobs, assembly times (again in hours) are distributed according to the Tria(1,3,8) distribution, while painting times are deterministic lasting 2 hours this model is simulated for 100,000 hours to obtain resource utilizations and mean flow times by job type, and average WIP (work-in-process) levels in each buffer

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 24

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 25

Dialog box for the StateSet module defining the states of each resource

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 26

Dialog box and spreadsheet view for the Record module tallying flow times

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 27

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 28

29

30

31

32

Theoretical utilizations at the Assembly workstation

the total workstation utilization is u = 0.9 the partial utilizations of type 1 jobs is u (1) = 2 / 4 = 0.5 the partial utilizations of type 2 jobs is u ( 2) = 4 / 10 = 0.4 the sum of utilizations checks, since

u = u (1) + u ( 2) = 0.5 + 0.4 = 0.9

the simulation results from the report estimate the utilization of the = 0.889 assembly workstation as u .

= 0.889 , are close, The two figures, u = 0.9 and u but would be even closer if we were to run the model longer

33

A verification via Littles formula is carried out by the following procedure:

the total arrival rate is l = 0.25 + 0.1 = 0.35 The simulation estimates of average flow time of type 1 and 2 jobs

S S

The estimated combined average flow time over all job types (weighted by the relative arrival rates) is

The average number of jobs in the system is estimated from the simulation run as the sum of average buffer sizes plus workstation utilizations over all job types, namely,

(k ) + u = [N (k )] = b

k

4.1456 + 4. 8111 + 0. 88924 + 0. 94098 = 10. 787 = 0. 35 31.122 = 10.893 = 10.787 and l W Thus, we find that N

agree closely, in agreement with Littles formula

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 34

Model validation

Validation activities are critical to the construction of credible models The standard approach to model validation is to collect data (parameter values, performance metrics, etc.) from the system under study and compare them to their model counterparts

the data collection effort of Input Analysis can provide the requisite data from the system under study

data collected is classified into input values and corresponding output values

35

Consider a generic system

Raw input data is obtained by observing the system over a time period

A model is constructed from the observed data and run to produce simulation data (that serve as input data sets) from which the corresponding output data (performance metrics) are then estimated

The correspondence of the generic inputs, I , and generic outputs, O , k k is shown below

Period 1 2 Input Data Set I1 I2 Output Metric Set O1 I2

M

N

M

IN Generic input and output data

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8

M

ON

36

Each set of output metric values constitutes a sample of random values (say, daily mean delay times) two sets (samples) of output daily mean delay data are collected for days i = 1K , N

1. a sample {D , K , D } of observed daily mean delays, 1 N

collected from the real-life system under study 2.

,K , D } of estimated daily mean delays, a sample { D 1 N

collected from runs of the simulation model of the system under study

Thus,

for the real-life system under study, O i = D i , i = 1, K , N

, i = 1, K , N for the simulation runs, O = D i i

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 37

To test that the observed and simulated delays have the same means, , i = 1, K , N , define the statistics G i = D i - D i assumed approximately normally distributed with mean m G 2 and variance s G H0 : m =0 G Perform hypothesis testing of the form H : m 0 G 1 G- m G with the test statistic t N - 1 = , where SG / N t N - 1 has a t distribution with N - 1 degrees of freedom G and S are the sample mean and sample standard deviation,

G

The null hypothesis cannot be rejected if the confidence interval above contains 0, supporting the model is valid

Altiok / Melamed Simulation Modeling and Analysis with Arena Chapter 8 38

- Chapter IV Sri bisa dijaikan bahasanUploaded byKeadilan Harus Ditegakkan
- Computational Social Science and System DynamicsUploaded bysapane
- MBA First & Second Year Syllabus - Sub Centre LaturUploaded bythakareyn
- Six SigmaUploaded byarumil96659
- QNT 273 Academic Success/snaptutorialUploaded byjakartabc22
- Study on Gis Simulated Water Quality ModelUploaded byprjpublications
- SimulationModelsUploaded byVelmurugan Rajarathinam
- Project RubricUploaded bymsnair
- qch9-10Uploaded bylightknow
- Modeling of Biological Manufacturing Systems for Dynamic Reconfiguration_uremovicUploaded byzagi1985
- Sampling DistributionsUploaded by03435013877
- 61041Uploaded byAfaf
- Biostatistics (Dr Shilpi Gilra)Uploaded byJitender Reddy
- Solutions Manual Probability and Statistics for Engineers and Scientists Manual HaylerUploaded byAmy Parker
- Perverse Nature of SDUploaded byaconnel
- Zipper Merge Evaluation of the Late Merge Traffic Work Zone 05-r6Uploaded bywilliam_V_Lee
- Lazear, Rosen ('81) - Rank Order TournamentsUploaded bymert0723
- Conditional Simulations of Fluid Flow in Three-dimensional Networks of Discrete FracturesUploaded byali
- 122665_Lecture 7a Synthesis PhaseUploaded bySivanesh Kumar
- Operation Management.pdfUploaded byShrikant Dash
- Std Deviation QuizUploaded byabhisek
- 013_DSETarget2Uploaded by918goody
- Biksu Word KongUploaded byyauri dp
- data analysisUploaded byWei Du Le
- Chapter TestmeasureofvariationUploaded byedniel maratas
- Report 2Uploaded byjohandreher
- DECSCI2-LABORATORY-WORK-1_-TEST-ON-ONE-POPULATION.docxUploaded byKirstenChua
- colgenUploaded byPervez Ahmad
- PRMO2016_answers.pdfUploaded byNimai Roy
- statistics.docxUploaded byjamee

- Statistical TablesUploaded byAlbertoMartinez
- T TestUploaded byMc Ortega
- A. Gibbs, J. Rosenthal. Statistics - Making Sense of DataUploaded byzabihicac
- Blackboard CLTUploaded byYiLong Xu
- Statistics- Tests of Hypotheses-1Uploaded byDr Rushen Singh
- Norm Dist...................Uploaded bylisahidayati26
- ch10.pptUploaded byPrince
- ch16.ciUploaded byamisha2562585
- Stat (Advanced) DrillsUploaded byHazel Marie Ignacio Peralta
- ST104a VleUploaded byletuan2212
- Hypothesis testing, Test Statistic (z,p,t,F)Uploaded bymarketingmixfourp
- ijest-ng-vol3-no8-pp1-12Uploaded byAnonymous iSsmdc
- Statistics-hypothesis Test Assignment HelpUploaded byExpertsmindEdu
- Assignment 2- StatisticUploaded byZuety Ikhsan
- Chapter 09Uploaded byMuneeb_2k
- E2206L0409bUploaded byAlicia Sun
- Finals Exam- SeedUploaded byDavy Rose
- Garch23 TutorialUploaded byHoracio Moyano Borquez
- Tutorial 1 Qamii Solutions 2016Uploaded byHafiz Yaakop
- 19659chapter-14Uploaded byKrutika Desai
- Chapter 2 Hypothesis TestingUploaded byMario M. Ramon
- z_and_t_testsUploaded bycintoy
- STA2023Ch09Uploaded byMohamed Med
- J Chem Edu 2008 85 1083Uploaded byffsodre
- LecturesUploaded byJones Ofei Agyemang
- Pearson Statistics Chapter 8 ToolsUploaded bySean Callahan
- On the Behavior of Dynamic Random VariablesUploaded byHugo Hernández
- (Boos, Hughes-Oliver) Applications of Basu's TheoremUploaded byJohann Sebastian Claveria
- A Bayesian Reliability Approach To Multiple Response Optimization With Seemingly Unrelated Regression Models_PaperUploaded byIntan Fitri Maharany
- Business StatisticsUploaded byRaktim