Modeling and Forecasting Forecasting Methods Measuring the Accuracy of Forecast

FORECASTING AND MODEL SELECTION

Anurag Prasad
Department of Mathematics and Statistics Indian Institute of Technology Kanpur, India

REACH Symposium, March 15-18, 2008

Forecasting and Model Selection

Modeling and Forecasting Forecasting Methods Measuring the Accuracy of Forecast

Outline

Modeling and Forecasting

Forecasting Methods

Measuring the Accuracy of Forecast

Forecasting and Model Selection

Modeling and Forecasting Forecasting Methods Measuring the Accuracy of Forecast

Assumptions of Forecasting

Element of Uncertainty Blind Spots Change in Forecast Accuracy

Forecasting and Model Selection

Modeling and Forecasting Forecasting Methods Measuring the Accuracy of Forecast

Framework of a Forecast System

ModelBuilding Phase
Theory and/or Historical Data

Forecasting Phase

Model Specification

Model Estimation

No Diagnostic Checking Yes

No Forecast Generation Stability Checking Yes New Observations Forecast Updation

Data

Forecasting and Model Selection

Modeling and Forecasting Forecasting Methods Measuring the Accuracy of Forecast

Choice of a Particular Forecast Model

Degree of Accuracy Required Cost of Producing Forecasts Forecast Horizon Degree of Complexity Required Available Data

Forecasting and Model Selection

Modeling and Forecasting Forecasting Methods Measuring the Accuracy of Forecast

Classication of Estimation Methods

Time Series Methods Causal Methods Judgemental Methods

Forecasting and Model Selection

Modeling and Forecasting Forecasting Methods Measuring the Accuracy of Forecast

Time Series Methods

Use historical data as a basis Underlying patterns are fairly stable
1 2 3 4 5 6 7

Autoregressive Moving Average (ARMA) Exponential Smoothing Extrapolation Linear Prediction Trend Estimation Growth Curve Box-Jenkins Approach

Forecasting and Model Selection

Modeling and Forecasting Forecasting Methods Measuring the Accuracy of Forecast

Causal Methods

Belief that some other time series can be useful Assumption that it is possible to identify the underlying factors
1

Regression Analysis * Linear Regression * Non-Linear Regression Econometrics

Forecasting and Model Selection

Modeling and Forecasting Forecasting Methods Measuring the Accuracy of Forecast

Judgemental Methods
Incorporate intuitive judgements, opinions and probability estimates
1 2 3 4 5 6

Composite Forecasts Surveys Delphi Method Scenario Building Technology Forecasting Forecast by Analogy

Forecasting and Model Selection

Modeling and Forecasting Forecasting Methods Measuring the Accuracy of Forecast

Forecast Error
For t = 1, . . . , N, y (t) : Actual value at period t, y(t) : Forecast value at period t; e(t) : Forecast error at period t;

e(t) = y (t) y (t)

y(t)

^ ) y(t i y(t ) i t t i

10

Forecasting and Model Selection

Modeling and Forecasting Forecasting Methods Measuring the Accuracy of Forecast

Graphical Measures of Forecast Accuracy

Plot of y (t) versus y (t) Keep the same scale for both the axes. Departure of points from the 450 line through origin indicates imperfect forecasts.

11

Forecasting and Model Selection

Modeling and Forecasting Forecasting Methods Measuring the Accuracy of Forecast

y(t)

y(t)

^ y(t) Correct Model Form

^ y(t) Incorrect Model Form

12

Forecasting and Model Selection

Modeling and Forecasting Forecasting Methods Measuring the Accuracy of Forecast

Plot of e(t) versus t Reveals patterns of variability which the model has failed to explain. For a good model, the forecast errors should vary in a horizontal band around zero.

13

Forecasting and Model Selection

Modeling and Forecasting Forecasting Methods Measuring the Accuracy of Forecast

e(t)
0

e(t)
0

t Correct Model Form

t Incorrect Model Form

e(t)
0

e(t)
0

t Incorrect Model Form

t Incorrect Model Form

14

Forecasting and Model Selection

Modeling and Forecasting Forecasting Methods Measuring the Accuracy of Forecast

Descriptive Measures of Forecast Accuracy

Descriptive Measures of Forecast Accuracy are used to...

1

Provide a single, easily interpreted measure of models reliability Compare the accuracy of two different models Search for an optimal model Monitor a models performance

2 3 4

15

Forecasting and Model Selection

Modeling and Forecasting Forecasting Methods Measuring the Accuracy of Forecast

Descriptive Measures of Forecast Accuracy

Mean Absolute Deviation (MAD) MAD =

|forecast error| number of forecasts

N t=1

|e(t)| N

Mean Square Error (MSE) MSE = =

(forecast error)2 number of forecasts

Root Mean Square Error (RMSE) RMSE = MSE

N t=1

e(t)2 N

16

Forecasting and Model Selection

Modeling and Forecasting Forecasting Methods Measuring the Accuracy of Forecast

Descriptive Measures of Forecast Accuracy

Mean Absolute Percent Error (MAPE) MAPE = =
N t=1

|forecast error/actual value| .100% number of forecasts

|e(t)/y (t)| .100% N

Pearsons Correlation Coefcient (r ) between y (t) and y (t)

r=

N t=1 (y (t)y )(y (t)y) N N 2 2 t=1 (y (t)y ) t=1 (y (t)y)

17

Forecasting and Model Selection

Modeling and Forecasting Forecasting Methods Measuring the Accuracy of Forecast

Descriptive Measures of Forecast Accuracy

"No Change" model is : y (t + 1) = y (t) Theils Inequality Coefcient (U) U=

RMSE("new" model) RMSE("no change" model)

U>1 U=1 U<1

worse than "no change" model as good as "no change" model better than "no change" model

18

Forecasting and Model Selection

Modeling and Forecasting Forecasting Methods Measuring the Accuracy of Forecast

Further Readings

Quantitative Forecasting Methods, N.R. Farnum and L.W. Stanton, 1989, PWS-KENT Publishing Co. Statistical Methods for Forecasting, B. Abraham and J. Ledolter, 1983, John Wiley & Sons Introduction to Time Series and Forecasting, P.J. Brockwell and R.A. Davis, 2002 (Second Edition) , Springer-Verlag Time Series Analysis and Forecasting, R. Yaffee, 2000, Academic Press

19

Forecasting and Model Selection

Modeling and Forecasting Forecasting Methods Measuring the Accuracy of Forecast

Thank You

20

Forecasting and Model Selection