Goodness of Fit

Outline

n Goodness of Fit
n Squared Error
n Chi-Square Statistic

2
 Readings

n Kelton et al. – Chapter 4.6

n Law -- Section 6.4.1, 6.4.2

3
 Goodness of Fit

n We collect data (Step 1)

n We postulate a distribution which we
think this data follows (Step 2)
n How do we know this distribution is
“correct”?
will use the term “fit”
will measure the quality of the fit

4
 Example
n Assume we observed the # of people
obtaining grades A, B, C, D, E last year in
Simulation class:
A B C D E
Observed # 20 15 12 3 0

Previous instructor tells us that he expects the

grade to follow a discrete uniform distribution 5
 Goodness of Fit
n How can we tell if our data supports the
previous instructor’s assumption about the
distribution?
n Need a way to compare the assumed
(postulated) probability distribution and the
distribution of the collected data

We will start to do this comparison

using the concept of squared error
6
 Squared Error Example

A B C D E
Observed # 20 15 12 3 0
Postulated probability 0.2 0.2 0.2 0.2 0.2
Expected # 10 10 10 10 10

Expected # of points falling into interval A

under the postulated distribution: 50(0.2)
7
 Squared Error Example

A B C D E
Observed # 20 15 12 3 0
Postulated probability 0.2 0.2 0.2 0.2 0.2
Expected # 10 10 10 10 10

Need to compare observed and expected!

So we take the difference between them and square it
8
 Squared Error

n ∑$
!"#(𝑁! − 𝑛𝑝! )
%

(Total # of observations in your data) x

(postulated theoretical probability of data
falling into interval i) = expected # in
Sum over all intervals/bins interval i under theoretical distribution

# of observations in bin I from the

data (i.e., observed #)

9
 Squared Error Example
n (20 − 10)! +(15 − 10)! +(12 − 10)!
+(3 − 10)! +(0 − 10)! = 278

Is this error high or low? We can’t really tell!

What error do we get if we try a different
distribution?

10
 Squared Error Example 2

n Assume we observed the # of people

obtaining grades A, B, C, D, E last year in
Simulation class:
A B C D E
Observed # 20 15 12 3 0

Previous instructor tells us that he expects the

following distribution 0.4, 0.2, 0.2, 0.2, 0
11
 Squared Error Example 2
n Assume a different postulated
probability distribution
A B C D E
Observed # 20 15 12 3 0
Postulated probability 0.4 0.2 0.2 0.2 0
Expected #

12
 Squared Error Example 2

A B C D E
Observed # 20 15 12 3 0
Postulated probability 0.4 0.2 0.2 0.2 0
Expected # 20 10 10 10 0

13
 Squared Error Example
n (20 − 10)! +(15 − 10)! +(12 − 10)!
+(3 − 10)! +(0 − 0)! = 78

Smaller is better so here

we get much better fit!

14
 Two Issues with Squared Error

n Scale: affected by the number of

observations

n Quality: how do we know the square

error is small enough?

15
Solution to Scale Issue

n Compute Chi-Square statistic

% " !
!
(𝑁 − 𝑛𝑝" )
𝑋 =#
𝑛𝑝"
"#$

16
 Solution to Quality Issue

n Use a formal statistical test which tells us

whether we have evidence to accept the
proposed distribution or not
n i.e., evidence of whether the distribution
estimated from the data is close to a
postulated theoretical distribution or not

17
 Squared Error Example 3
n Given the following data: 102, 112, 131,
107, 114, 95, 133, 145, 139, 117

n I say it follows a uniform distribution

U[95, 150]

n Find the squared error and chi-square-

statistic

18
 Squared Error Example 3

[95,107.5) [107.5,120) [120,132.5) [132.5,150]

Observed # 3 3 1 3
Post. prob. 0.25 0.25 0.25 0.25
Expected # 2.5 2.5 2.5 2.5

Square-Error = 3, Chi-Square Statistic = 1.2

19
 Chi-Square Test
n Main idea: evaluate how extreme this Chi-
Square statistic is!

n If it is too extreme, then we have evidence

that the postulated distribution is not correct

n If it is not too extreme, then we do not

have such evidence

20
 Chi-Square Test
n Null hypothesis
n 𝐻" : The observed data is following the
postulated distribution
n We need to evaluate
P ( observing a value as large or greater than
the current chi-square statistic | 𝐻& is true )
=P(𝑋 ! ≥t)=p-value
If low (typically ≤ 0.05), then
we have evidence to reject H0 21
 '
$ #
(𝑁 − 𝑛𝑝$ )
𝑋# = $
Chi-Square Test $%&
𝑛𝑝$

P ( observing a value as large or greater than

the current chi-square statistic | 𝐻" is true )
=P(𝑋 ! ≥t)=p-value
≈ 𝑃(𝜒!" ! ≥ 𝑡) [for large values of n]
where 𝜒#$ ! is a random variable distributeḑ
according to the Chi-Square distribution with df
degrees of freedom

22
 Chi-Square Distribution
Degrees of
freedom

https://commons.wikimedia.org/wiki/File:Chi-square_pdf.svg 23
Chi-Square Table

24
 So how do we use the Chi-
Square distribution?
n Assume a significance level, alpha
n Frequently assumed 0.05

n Determine degrees of freedom

n df = numBins – 1 (this is a conservative choice)

Input analyzer uses a slightly different way

of calculating the degrees of freedom
25
 Squared Error Example 3

[95,107.5) [107.5,120) [120,132.5) [132.5,150]

Observed # 3 3 1 3
Post. prob. 0.25 0.25 0.25 0.25
Expected # 2.5 2.5 2.5 2.5

Chi-Square Statistic = 1.2, DF= 4-1=3

26
 Method 1: Critical Value
n Determine the Chi-Square critical value for
the given alpha and df
n If Chi-Square statistic > critical value,
reject null hypothesis
n Otherwise, do not have evidence to reject
null hypothesis

Is our observed Chi-Square “too extreme”?

27
 Chi-Square Statistic = 1.2,
DF = 4 – 1 = 3
Example 3 alpha = 0.05

Chi-square = 1.2 not > critical value = 7.82,

so we do not reject H0 28
 Chi-Square Distribution

observed Critical Value

https://commons.wikimedia.org/wiki/File:Chi-square_pdf.svg 29
 Method 2: P-Value

n Determine the p-value and compare it to

alpha

n If p < alpha, reject null hypothesis

n Otherwise, do not have evidence to
reject null hypothesis

Is the probability of observing our

value under H0 “too small”?
30
 Chi-Square Statistic = 1.2,
DF = 4 – 1 = 3
Example 3 alpha = 0.05

P-value ≈ 0.75 > alpha = 0.05, so we do not reject H0

31
 Chi-Square Distribution

observed P-value ≈ 0.75

https://commons.wikimedia.org/wiki/File:Chi-square_pdf.svg 32
 Steps in Chi-Square Test

1.Calculate Chi-Square Test statistic

2.Find df

3.Determine alpha

4.Use method 1 or method 2 to determine

whether H0 should be rejected

33
 Steps in Input Analysis
1. Data Collection
2. Data Analysis
a) Plots (especially histogram!)
b) Summary Statistics & Observations
c) Independence Tests
3. Fitting a Distribution
4. Goodness of Fit (GOF) Tests
5. Return to Step 1, 2 or 3 ifGOFispoor!

34
 CAUTION!
n Input Analyzer cannot always give us
conclusive evidence
n What if the p-value of the best fit
distribution is very low?

n Input analyzer cannot detect problems

with your data such as outliers or
dependence

35
 So…
n You need to perform all steps
previously described!

n Specifically, need to use summary

statistics, observations & knowledge of
system together with Arena Input
Analyzer, not just rely on the tool itself!

36
Questions?

37