You are on page 1of 22

Evaluation

The evaluation of Naive Forecasting Techniques
relies primarily on the comparison of the forecasts
with the corresponding actual values
Evaluation Methods

÷Mean Error (ME)
÷Mean Absolute Error (MAE)
÷Mean Squared Error (MSE)
÷Mean Percentage Error (MPE)
÷Mean Absolute Percentage Error (MAPE)

The ME can be very misleading. A ME value of
zero can mean that the method forecasted the
actual values perfectly (unlikely) or that the
positive and negative errors cancelled each
other out. It tends to Understate the error in
all cases.
( ) n F A ME
n
i
t t
/
1
|
.
|

\
|
÷ =
¿
=
Evaluation Methods

÷Mean Error (ME)
÷Mean Absolute Error (MAE)
÷Mean Squared Error (MSE)
÷Mean Percentage Error (MPE)
÷Mean Absolute Percentage Error (MAPE)
MAE is a way of dealing with the Understatement
of ME. By using the Absolute values of the error,
the mean gives a better indication of the model’s fit.
MAE A F n
t t
i
n
= ÷
=
¿
( ) /
1
Evaluation Methods

÷Mean Error (ME)
÷Mean Absolute Error (MAE)
÷Mean Squared Error (MSE)
÷Mean Percentage Error (MPE)
÷Mean Absolute Percentage Error (MAPE)
The MSE eliminates the positive/negative
problem by squaring the errors. The result
tends to place more emphasis on the larger
errors and, therefore, gives a more conservative
measure than the MAE.


( )
MSE A F n
t t
i
n
= ÷
|
\

|
.
|
=
¿
2
1
/
The previous three measures are “series
specific;” i.e., they only allow evaluation of
the series that generated the errors.

The next two measures, by using the
percentage of the error relative to the actual,
are designed to allow comparison of the
results with different models.
Evaluation Methods

÷Mean Error (ME)
÷Mean Absolute Error (MAE)
÷Mean Squared Error (MSE)
÷Mean Percentage Error (MPE)
÷Mean Absolute Percentage Error (MAPE)
The MPE is a relative measure of the forecasting
error. It is subject to the “averaging” of the positive
and negative errors.
MPE
A F
A
n
t t
t
t
n
=
÷ |
\

|
.
|
×
|
\

|
.
|
=
¿
1
100 /
Evaluation Methods

÷Mean Error (ME)
÷Mean Absolute Error (MAE)
÷Mean Squared Error (MSE)
÷Mean Percentage Error (MPE)
÷Mean Absolute Percentage Error (MAPE)
MAPE is a comparative measure that does not have
the problem of averaging the positive and negative
errors. It is relatively easy to use to communicate a
model’s effectiveness.
MAPE
A F
A
n
t t
t
t
n
=
÷
|
\

|
.
|
×
|
\

|
.
|
=
¿
100
1
/
Measurement of Forecasting Error
Mean Error (ME): The average of all the errors of
forecast for a group of data.
Mean Absolute Error (MAE): The mean, or average
of the absolute values of the errors.
Mean Square Error (MSE): The average of the
squared errors.
Mean Percentage Error (MPE): The average of the
percentage errors of a forecast.
Mean Absolute Percentage Error (MAPE): The
average of the absolute values of the percentage errors
of a forecast.
Example:
Nonfarm
Partnership
Tax
Returns:
Actual and
Forecast
with o = .7
Year Actual Forecast Error
1 1402
2 1458 1402.0 56.0
3 1553 1441.2 111.8
4 1613 1519.5 93.5
5 1676 1584.9 91.1
6 1755 1648.7 106.3
7 1807 1723.1 83.9
8 1824 1781.8 42.2
9 1826 1811.3 14.7
10 1780 1821.6 -41.6
11 1759 1792.5 -33.5
t t t F A F ) 1 ( ) ( 1 o o ÷ + = +
Mean Error
for the Nonfarm Partnership Forecasted Data
ME
i
e
=
=
=
¿
number of forecasts
524 3
10
52 43
.
.
Year Actual Forecast Error
1 1402.0
2 1458.0 1402.0 56.0
3 1553.0 1441.2 111.8
4 1613.0 1519.5 93.5
5 1676.0 1584.9 91.1
6 1755.0 1648.7 106.3
7 1807.0 1723.1 83.9
8 1824.0 1781.8 42.2
9 1826.0 1811.3 14.7
10 1780.0 1821.6 -41.6
11 1759.0 1792.5 -33.5
524.3
Mean Absolute Error
for the Nonfarm Partnership Forecasted Data
MAD
i
e
=
=
=
¿
number of forecasts
6745
10
67 45
.
.
Year Actual Forecast Error |Error|
1 1402.0
2 1458.0 1402.0 56.0 56.0
3 1553.0 1441.2 111.8 111.8
4 1613.0 1519.5 93.5 93.5
5 1676.0 1584.9 91.1 91.1
6 1755.0 1648.7 106.3 106.3
7 1807.0 1723.1 83.9 83.9
8 1824.0 1781.8 42.2 42.2
9 1826.0 1811.3 14.7 14.7
10 1780.0 1821.6 -41.6 41.6
11 1759.0 1792.5 -33.5 33.5
674.5
E
Mean Square Error
for the Nonfarm Partnership Forecasted Data
MSE
i
e
=
=
=
¿
2
55864 2
10
5586 42
number of forecasts
.
.
Year Actual Forecast Error Error
2
1 1402
2 1458 1402.0 56.0 3136.0
3 1553 1441.2 111.8 12499.2
4 1613 1519.5 93.5 8749.7
5 1676 1584.9 91.1 8292.3
6 1755 1648.7 106.3 11303.6
7 1807 1723.1 83.9 7038.5
8 1824 1781.8 42.2 1778.2
9 1826 1811.3 14.7 214.6
10 1780 1821.6 -41.6 1731.0
11 1759 1792.5 -33.5 1121.0
55864.2
Mean Percentage Error
for the Nonfarm Partnership Forecasted Data
MPE
i
i
e
X
=
-
|
\

|
.
|
=
=
¿
100
318
10
318%
number of forecasts
.
.
Year Actual Forecast Error Error %
1 1402
2 1458 1402.0 56.0 3.8%
3 1553 1441.2 111.8 7.2%
4 1613 1519.5 93.5 5.8%
5 1676 1584.9 91.1 5.4%
6 1755 1648.7 106.3 6.1%
7 1807 1723.1 83.9 4.6%
8 1824 1781.8 42.2 2.3%
9 1826 1811.3 14.7 0.8%
10 1780 1821.6 -41.6 -2.3%
11 1759 1792.5 -33.5 -1.9%
31.8%
Mean Absolute Percentage Error
for the Nonfarm Partnership Forecasted
Data
MAPE
i
i
e
X
=
-
|
\

|
.
|
|
=
=
¿
100
40 3
10
4 03%
number of forecasts
.
.
Year Actual Forecast Error |Error %|
1 1402
2 1458 1402.0 56.0 3.8%
3 1553 1441.2 111.8 7.2%
4 1613 1519.5 93.5 5.8%
5 1676 1584.9 91.1 5.4%
6 1755 1648.7 106.3 6.1%
7 1807 1723.1 83.9 4.6%
8 1824 1781.8 42.2 2.3%
9 1826 1811.3 14.7 0.8%
10 1780 1821.6 -41.6 2.3%
11 1759 1792.5 -33.5 1.9%
40.3%

Use of Error Measures
To identify the best forecasting method
• Use error measure to identify the best value
for the parameters of a specific method.
• Use error measure to identify the best
method.
• Use MSE and MAE for both of these
situations. Note that MSE tends to
emphasize large errors.



Use of Error Measures, continued

Forecast bias is the tendency of a
forecasting method to over or under predict.

The mean error, ME, measures the forecast
bias.