IE306

SYSTEMS SIMULATION

Ali Rıza Kaylan

kaylan@boun.edu.tr

1
 LECTURE 9
OUTLINE

Lifecycle of a Simulation Project

Model Verification
Model Validation
Validation Techniques
Subjective
Statistical
M/M/1 Example

2
 LIFECYCLE OF A SIMULATION PROJECT

VERIFICATION VALIDATION

Conceptual Simulation Simulation

Model Program Output

VALIDATION CREDIBILITY

System Implementation

MODEL CREDIBILITY: When a simulation

model and its results are accepted by the model
user as being valid and are used as a decision aid. 3
 MODEL VERIFICATION

Checking the transformation of the conceptual

simulation model into a computer program
with sufficient accuracy.
BUILDING THE MODEL RIGHT

VERIFICATION

Conceptual Simulation

Model Program

4
 MODEL VALIDATION

Determining whether the conceptual

simulation model is an accurate
representation of the system under study.
BUILDING THE RIGHT MODEL
(I ,I ,..., I )
1
s s
2
s
n
(I
1
m
, I2 ,..., In )
m m

INPUT INPUT
VARIABLES VARIABLES
(P , P ,..., P )
1
s s
2 l
s
(P1
m m
,P2 ,..., Pl
m
)
SYSTEM MODEL
PARAMETERS PARAMETERS
RESPONSE RESPONSE
VARIABLES VARIABLES
(R , R ,..., R )
1
s s
2
s
k (R
1
m
,R2m ,..., Rkm ) 5
 VALIDATION TECHNIQUES
SUBJECTIVE STATISTICAL
Event Validation Analysis of Variance
Face Validation Confidence Intervals
Field Tests Factor Analysis
Graphical Comparisons Hotelling's T2 Tests
Hypothesis Validation Multivariate Analysis
Predictive Validation Nonparametric Tests
Sensitivity Analysis Regression Analysis
Submodel Testing Theil's Inequality Coefficient
Turing Test Time Series Analysis
t-tests 6
 VALIDATION TECHNIQUES

EVENT VALIDATION
Employs identifiable events or event patterns as
criteria against which to compare model and
system behaviours.
FACE VALIDATION
People knowledge about the system under
study, based upon their estimates and intuition,
compare model and system behaviours to judge
whether the model and its results are
reasonable.

7
 VALIDATION TECHNIQUES

FIELD TESTS
The model is put in an operational
situation for the purpose of collecting as
much information as possible for model
validation.
GRAPHICAL COMPARISONS
The graphs of model and system
variables are compared to investigate
similarities and discrepancies.
Characteristics such as periodicities,
skewness, trend lines, inflection points
are checked. 8
 VALIDATION TECHNIQUES

HYPOTHESIS VALIDATION
Hypothesized input output relationships
for the system under study and the
developed model are compared.
PREDICTIVE VALIDATION
The model is driven by past system input
data and its forecasts are compared with
the corresponding past system output
data to test the predictive ability of the
model.
9
 VALIDATION TECHNIQUES

SENSITIVITY ANALYSIS
Performed by systematically changing
the values of model input variables and
parameters over some range of interest
and observing the effect upon model
behaviour.

10
 VALIDATION TECHNIQUES

TURING TEST
1. Find experts about the system under study,
2. Present them with two sets of output data one
from the model and one from the system
obtained under the same input conditions,
3. Without identifying which one is which, ask
them to differentiate between the two,
4. If they succeed, get feedback for correcting
model,
5. If they can not differentiate, our confidence in
model validity is increased.
!
11
 VALIDATION TECHNIQUES

Impose some additional assumptions so that

analytical solution exists.
Simulate this model and compare results with the
analytical solution.

Example: GI/G/1 Workstation represented as

M/M/1 QUEUEING SYSTEM

12
 VALIDATION TECHNIQUES

M/M/1 QUEUEING SYSTEM

ANALYTICAL SOLUTIONS
!
1
E(W ) =
µ−λ E(N ) = λ * E(W )
λ E(N q )= λ * E(W q )
E(Wq ) =
µ (µ − λ )

E(W ) = E(W q )+ E(S)

#1− ρ t=0
F(t) = $
%1 − ρ * exp[−µ (1 − ρ )t ] t> 0 13
 VALIDATION TECHNIQUES

M/M/1 QUEUEING SYSTEM

EXAMPLE
! 1
E(Interarrival Time) =10 minutes E(W ) = = 10
0.2 − 0.1
λ = 0.1 arrivals / minute 0.5
E(Wq ) = =5
E(Service Time) = 5 minutes 0.2 − 0.1
µ = 0.2 services / min ute
E(N ) = 0.1*10 = 1
ρ = λ / µ = 0.5
E(N q )= 0.1* 5 = 0.5
14
 VALIDATION TECHNIQUES
M/M/1 QUEUEING SYSTEM EXAMPLE
! Service
Arrival
No IA S A SS SE Wq W I ID
1 9 2 9 9 11 0 2 9 11
Mean 10 5 2 1 21 9 11 32 0 22 0 21
3 6 3 16 32 35 16 19 0 3
4 5 5 21 35 40 13 19 0 5
Simulation End Time= 989 5 10 8 32 40 47 8 16 0 8
Server Utilization= 0.515 6 6 1 37 47 49 10 12 0 1
7 26 5 63 63 68 0 5 15 19
8 5 8 69 69 76 0 8 1 9
9 4 8 73 76 85 4 12 0 8
IA 9.84 10 1 2 74 85 87 11 13 0 2
11 4 0 78 87 87 9 9 0 0
S 5.09 12 40 2 118 118 120 0 2 31 32
13 6 28 124 124 152 0 28 4 32
Wq 4.531 14 16 1 140 152 153 12 13 0 1
15 24 2 164 164 167 0 2 11 13
W 9.634 16 5 1 169 169 171 0 1 3 4
17 8 1 177 177 178 0 1 7 7
Lq 0.458 18 18 1 195 195 196 0 1 17 18
19 12 17 207 207 225 0 17 11 29
L 0.974 20 6 4 214 225 228 11 15 0 4

ID 9.89 99 3 6 976 977 983 1 7 0 6

100 8 5 984 984 989 0 5 1 6
15
 VALIDATION TECHNIQUES
M/M/1 QUEUEING SYSTEM EXAMPLE
!

E(W) E(Wq) E(N) E(Nq)

System 10 5 1 0.5
Model 9.634 4.531 0.974 0.458

16
 VALIDATION TECHNIQUES
M/M/1 QUEUEING SYSTEM EXAMPLE
!
Experiment 1 2 3 4 5 Mean St.Dev.
End Time 989 1103 973 1067 1096 1046 61
Utilization 0.515 0.471 0.52 0.474 0.43 0.482 0.037
IA 9.84 11 9.68 10.57 10.95 10.41 0.62
S 5.09 5.19 5.06 5.06 4.71 5.02 0.18
Wq 4.531 5.435 5.317 2.895 3.471 4.33 1.122
W 9.634 10.647 10.373 7.966 8.186 9.361 1.233
Lq 0.458 0.493 0.546 0.271 0.317 0.417 0.118
L 0.974 0.965 1.066 0.747 0.747 0.9 0.145
ID 9.89 11.03 9.73 10.67 10.96 10.46 0.61

17
 VALIDATION TECHNIQUES
M/M/1 QUEUEING SYSTEM EXAMPLE
CONFIDENCE INTERVAL (α=5%) for W
!
s2
X ± t n-1, 1-α /2
n
1.2332
9.361 ± t 4, 0.975
5
1.2332
9.361 ± 2.776
5
9.361 ± 1.531
[7.830,10.892 ] 18
 VALIDATION TECHNIQUES
THEIL’S INEQUALITY COEFFICIENT
!
Measure of the degree to which
the time series obtained from the system agrees with
the time series obtained from the simulation model.
n
2
" mi si / n
(Y ! Y )
i =1
U =
n n
2 2
"Y mi
/n + "Y si
/n
i =1 i =1

U=1 No agreement
U=0 Perfect agreement
19