Neural Network Models for Forecast: A Review

Leorey Marquez*, Tim Hill*, Marcus OConnor**, and William Remus*

* University

of Hawaii

** University

of New South Wales

Abstract
Neural networks are advocated as a replacement for statistical forecasting methods. In this paper we review the literature comparing neural networks and classical forecasting methods, particularly in causal forecasting, time series forecasting, and judgmental forecasting. We provide not only an overview and evaluation of the literature but akso summarize several studies we have performed which address the typical criticisms of work in this area. Overall, the empirical studies find neural networks at least as good as their classical counterparts.

Neural Networks and Time Series Forecasting Models

In time series forecasting, a variable is forecast based only on the history of that variable. For example, the number of visitors arriving in Hawaii next month might be made based only on the recent monthly data on visitor arrivals in Hawaii. S e i l methods have been developed pca for time series prediction including the moving average, exponential smoothing, and Box-Jenkins models. Neural networks and traditional time series techniques have been compared in at least seven different studies; the studies are briefly outlined in Exhibit 1. In general, these authors found neural networks and time series models to have roughly equivalent performance. The best of these studies have used the data from the well known M-competition (Makridakis et al., 1982). Makridakis et al. initially gathered 1001 real time series; they then selected a systematic sample of 111 series. In the original competition, various groups of forecasters were given all but the most recent data points in each set (the holdout sample) and asked to make time series forecasts for the most recent points. The competitors forecasts were then compared to the actual values in the holdout data set. The results of this competition were reported in Makridakis et al. (1982). Since the competition, all 1001series including the holdout data are easily available. Sharda and Patil(1990a) used 75 of the 111 series and found that neural network models performed as well as the automatic Box-Jenkins (autobox) procedure. They noted problems in the remaining 36 series. Foster, Collopy, and Ungar (1991) also used the M-competition data. They used all 111 series and found neural networks to be inferior to the best classical models from the Mcompetition. Using fewer and shorter series, Sharda and Patil (1990b) and Tang et al. (1990) agreed that for time series

K y Words: Neural Networks, Regression, Forecasting, e Judgmental Forecasting, Time Series. Introduction
Over the last few decades, there has been much research directed at understanding and predicting the future. This research has led to many developments in forecasting methods - including many methodological advances based on statistical techniques. Currently, there is a new challenger for these methodologies - neural networks. Neural networks (NN) have been widely touted as solving many forecasting problems. For example, they are argued to be able to model easily any type of parametric or non-parametric process including automatically and optimally transforming the input data. These sorts of claims have led to much interest in neural networks. In this paper we review the literature comparing neural networks and the classical forecasting techniques. Our review will be segmented into three different applications areas: time series forecasting, regression-based causal forecasting, and regression-based judgmental forecasting. Additionally, we will note the literature comparing neural networks and other linear models such as discriminant analysis and logit models.

494

0073-1129-1/92 $3.00 0 1992 IEEE

with long memory, neural network models and BoxJenkins models produce comparable results. For time series with short memory, they argued that neural networks outperform the Box-Jenkins models. When forecasting nonlinear dynamic systems with chaotic behavior, Lapedes and Farber (1987) showed that neural networks can be orders of magnitude more accurate than conventional methods. Neural networks have also been implemented for diagnostic applications in time series analysis. Sastri et al. (1990) compared the performance of backpropagation, counterpropagation, and categocy learning neural network models in identifying the auto-regressive time series models. The backpropagation network was best in identifying the correct time series model while the counterpropagation network was best in detecting changes in the model. Foster, Collopy, and Ungar (1991) had similar success in using neural networks to identify the best time series model. As part of the neural network sessions at Hawaii International Conference of System Sciences, the referees developed a list of desirable characteristics for such studies (see Exhibit 2). AU the above studies (except Foster, Collopy, and Ungar) can be faulted on one or more of the points in that list. Those studies not using Mcompetition data generally had far too few series and too few data points per series to support generalization; thus, they are only suggestive. The Sharda and Patil study (1990b) mysteriously threw away one third of the series; the latter leads us to question the generalizeability of their results. Foster, Collopy, and Ungar (1991) didnt have that problem but found neural networks to be inferior to the classical methods. In comparing forecasting models it is important to develop the model on one set of data and then test it on another. In neural network studies, the experimenters often develop many neural network models and compare them on the holdout data. Then they select the best to compare with the classical models. This may have happened in some of the above studies; that is, the models appeared partially fit on the holdout data (rather than using the holdout data just for the final test comparison). AU the concerns in this and the above prior paragraph led us to conduct the following experiment. In the &U, OConnor, and Remus (1991) research, time series forecasts based on neural networks were compared with forecasts from six statistical time series methods (including exponential smoothing and BoxJenkins) and two judgment-based methods. The experiment used all 111 time series from the Mcompetition. The classical methods were all estimated by experts in a particular technique as part of the Mcompetition.

The 111 series were each divided by Makridakis et al. into a portion of model estimation and a portion for model testing. The structure of the neural networks in this study was developed using the complete series #1 and #2; the remaining 109 neural networks used the structure found in #1 and #2 but were estimated by the first author using the same ground rules as the M-competition. That is, the first author was not supplied with the holdout data (except in the cases of series #1 and a). We then compared his neural network models forecasts for the holdout data with the forecasts made by experts in the M-competition with the classical methods. Amss all 109 series, the neural networks did significantly better or as good as statistical and human judgment methods.

Neural Networks and Regression-Based Casual Forecasting Models

Regression models are often used to make causal forecasts. In this kind of forecast, a variable is predicted by other variables that are causally linked to the former. Interestingly, no groups thus far have attempted to do this type of forecasting with neural networks. Therefore, we conducted the following simulation study. In Marquez et al. (1991), we generated data representing three common bivariate functional forms used in causal forecasting (linear, log-linear, and reciprocal) and then compared the performance of the neural network models against the true regression model. For each functional form, 100 sets of n (15, 30, or 60) points each with three noise levels (R2= .30, .a, .90) were created and to develop the alternative models. These were then tested on simulated holdout data. The results showed that neural network models perform within 2% of the mean absolute percentage error ( W E ) ; this is very good performance in the real world. In this study neural networks performed comparatively well in high noise and with relatively few data points. This work is continuing as Marquez expands the study to other issues such as relative vulnerability to multicolinearity, outliers, and other data problems.

Neural Networks and Judgemental Forecasting Models

A judgmental forecast tries to predict a humans judgment based on the factors that a person uses to made the judgment. Generally, these forecast models have linear functional forms and regression is used to estimate them. There are numerous interesting business decisions that have been examined using judgmental forecasting models; the list of comparative studies is shown in Exhibit 3.

495

Dutta and Shekhar (1988) used 10 factors to predict the ratings of corporate bonds. They found that the neural networks outperformed the regression models. Duliba (1991) compared neural network models with four types of regression models in predicting the financial performance of transportation companies. She found that the neural network model outperformed the random effects regression model but not the fixed effects model. As in time series comparisons, these studies also suffer from having a small number of data sets and few data points per set. Also, the holdout data may have effected model estimation and specification. To overcome these problems, Remus and Hill (1990) compared the production scheduling decisions as modeled by neural networks and regression-based decision rules. The data used was from 62 decision makers who each made 24 decisions (Remus, 1987); thus, neural network and regression models were developed and compared for each of the 62 decision makers. The model structure used was the same structure used in the comparable regression models. Neural network models performed as well but not better than those using the linear regression models. In a second study (Remus and Hill, 1991), we continued the above research and aggregated the data from all 62 decision makers to estimate a composite neural network modeL The resulting neural network model performed better than both the classical models and neural networks from the earlier study; the composite neural network also performed at least as well as classical composite models. It is important to note that the above two studies did not use holdout samples so that the procedures would parallel those used in the classical models.

analysis. Practical problems where the comparison between neural networks and disaiminant analysis has been applied include the prediction of stock price performance (Yoon and Swales, 1990), the prediction of company bankruptcy (Odom and Sharda, 1990; Raghupathi et al., 1991; Koster et al., 1990), and the assignment of ratings to bonds (Surkan and Singleton, 1990). In all of these studies, the neural network model outperformed discriminant analysis. Neural networks proved to be more robust (Odom and Sharda, 1990), were able to extract higher level features for better generalization (Raghupathi et al., 1991), and detected aspects of the data set space that were beyond the capability of the statistical model (Koster et al., 1990). These studies again suffer from many of the shortcomings listed in Exhibit 2. In particular, the number of data sets is limited and the size of the data sets is small. In many cases, the holdout was iteratively used to provide information to aid in selecting the best model; therefore, the models are partially fit on the holdout data. Thus, the studies in this area are suggestive but do not critically demonstrate the advantages of neural networks.

Summary
Across the areas of time series forecasting, regression forecasting, and judgmental forecasting, neural networks seem to perform at least as well as classical models. This is still and preliminary finding and in need of more exploration. In particular, few studies have examined the (1) data conditions or (2) implementation situations favoring neural networks. Marquez et al. (1991), as an example of the former, demonstrated that neural networks are more advantageous with low sample sizes and high levels of noise. As an example of the latter, Remus and Hill (1990 and 1991) argue for the advantages of neural networks when building intelligent systems. Also there is a need for rigorous studies of the relative robustness of neural networks and classical models. The current work by Marquez will begin to address these issues. Also, most of these studies use backpropagation; perhaps new and better models might improve neural network's performance.

Neural Networks and Other Linear Models

For completeness, this section will report other studies where linear models have been compared to neural networks in common business tasks, see Exhibit 4 for a summary. For example, logistic regression is commonly used in classification problems where the response variable has a binary value. Bell et al. (1989) compared backpropagation networks against logistic regression models in predicting commercial bank failures. The neural network model performed well in failure prediction and the expected costs for misclassification by the neural models were found to be lower than those of the logit model. Roy and Cosset (1990) also used neuml network and logistic regression models in predicting country risk ratings using economic and political indicators. The neural network models had lower mean absolute emf in their predictions and reacted more evenly to the indicators than their logistic counterparts. Neural networks are also an alternative to disaiminant

References
Bell, T., G. Ribar and J. Verchio, "Neural Nets vs. Logistic Regression," presented at the University of Southern California Expert Systems Symposium, November, 1989. Duliba, K., "Contrasting Neural Nets with Regression in Predicting Performance," Proceedines of the 24th Hawaii International Conference on System Sciences, Vol. 4, 163-170

496

(1991). Dutta, S . and S . Shekhar, "Bond Rating: A Non-Conservative Application of Neural Networks," Proceedings of the 1988 I " Vol. 2, 443-450 (1988). C. Foster, B., F. Collopy and L. Ungar, "Neural Network Forecasting of Short, Noisy Time Series," presented at the ORSA TIMS National Meeting, May, 1991. Koster, A., N. Sondak and W. Bourbia, "A Business Application of Artificial Neural Network Systems," The Journal of Computer Information Svstems, Vol. XXXI, 3-10 (1990). Lapedes, A. and R. Farber, "Nonlinear Signal Prediction Using Neural Networks: Prediction and System Modelling," Los Alamos National Laboratory Report LA-UR-87-2665 (1987). Makridakis, S., A. Anderson, R. Carbone, R. Fildes, M. Hibon, R. Lewandowski, J. Newton, E. Parzen, E. and R. Winkler, "The Accuracy of Extrapolation (Time Series) Methods: Results of a Forecasting Competition," Journal of Forecasting, 1, 111153 (1982). Marquez, L., T. Hill, W. Remus and R. Worthley, "Neural Network Models as an Alternative to Regression," Proceedinas of the 24th Hawaii International Conference on Svstem Sciences, Vol. 4, 129-135 (1991). Odom, M. and R. Sharda, "A Neural Network Model for Bankruptcy Prediction," Proceedings of the 1990 UCNN, Vol. 2, 163-168 (1990). Raghupathi, W., L. Schkade and R. Bapi, "A Neural Network Application for Bankruptcy Prediction," Proceedings of the 24th Hawaii International Conference on Svstem Sciences, Vol. 4, 147-155 (1991). Remus, W., "A Study of the Impact of Graphical and Tabular Displays and Their Interaction with Environmental Complexity," Management Science, Vol. 33, 1200-1205 (1987). Remus, W. and T. Hill, "Neural Network Models of Managerial Judgment," Proceedings of the 23rd Hawaii International Conference on Svstem Sciences, Vol. 4, 340-344 (1990). Remus, W. and T. Hill, "Neural Network Models for Intelligent Support of Managerial Decision Making," University of Hawaii Working Paper, (1991). Roy, J. and J. Cosset, "Forecasting Country Risk Ratings Using a Neural Network," Proceedinas of the U r d Hawaii International Conference on Svstem Sciences, Vol. 4, 327-334 (1990). Sastri, T., J. English and G. Wang, "Neural Networks for Time Series Model Identification and Change Detection," Texas A&M (1990). University Working Paper INEN/QC/WP/03/07-90

Sharda, R. and R. Patil, "Neural Networks as Forecasting Experts: An Empirical Test," Proceedings of the 1990 UCNN Meeting, Vol. 2, 491-494 (199Oa). Sharda, R. and R. Patil, "Connectionist Approach to Time Series Prediction: An Empirical Test," Oklahoma State University Working Paper 90-26 (1990b). Surkan, A. and J. Singleton, "Neural Networks for Bond Rating Improved by Multiple Hidden Layers," Proceedings of the 1990 UCNN, Vol. 2, 157-162 (1990). Tang, Z., C. de Almeida and P. Fishwick, "Time Series Forecasting Using Neural Networks vs. Box-Jenkins Methodology," presented at the 1990 International Workshop on Neural Networks, Feb., 1990. Yoon, Y.and G. Swales, "Predicting Stock Price Performance," Proceedings of the 24th Hawaii International Conference on System Sciences, Vol. 4, 156-162 (1991).

Exhibit 1 Literature Comparing Neural Networks and Time Series Methods Study Hill et al. (1991) Sastri et al. (1990) Type of Data 111 time series 92 simulated, 1 real time Results N"s were superior to all classical models. Backpropagation was the best model indentifier. Counterpropagation detected series model changes best. N"s are comparable in performance to Automatic Box-Jenkins. With short memory, NN outperformed Box-Jenkins. N"s are more accurate than conventional methods. N"s appear to be better than Box-Jenkins for long term forecasting. N"s were inferior to classical methods.

Sharda and Patil (199Oa) Sharda and Patil (199Ob)

75 time series

111 time series

Lapedes and Farber chaotic (1987) series Tang et al. (1990) Foster et al. (1991)
3 time series

111 time series

497

Exhibit 2 The HICSS Reviewers Summary Opinions on what Constituted an Excellent Neural Network (NN) Paper The following are the desirable attributes of a research paper on neural networks for a business application: 1 The business task modeled should be important. . 2 The literature on the task and on previous approaches to modeling . the task should be well reviewed. 3. The classical non-NN approaches to the task should be compared to the N N models. 4 Multiple sets of realistic data should be used for the evaluation. The . data should be such that we can feel certain that the comparison between Ns and the classical methods applies to a wide class of similar situations. For example, when using a forecasting task, the Ns and classical models should be compared on several different real data series.

5. The NN and classical models should be evaluated on a different set of data than that on which the N N was trained and the model estimated. 6. Statistics should be reported which not only show that 4s work but also how they compare to classical non-NN approaches. The statistics should include those normally used to evaluate the classical non-NN approaches plus any other critical measures. 7. Adequate detail should be reported on the NN implementation so othen can replicate the work. For example, the software used, values of options selected for the NN,and the epochs required should be reported.

Exhibit 3 Literature Comparing Neural Networks and Regression Analysis in Judgmental Forecasting Study Study Duliba Type of Data Results Bell

Exhibit 4 Literature Comparing Neural Networks and Other Linear Models Type of Data commercial bank failures bankruptcy data for 91 business organizations country risk rating Results

8-10-2 8-14-2 and NNs

had lower misclasification costs than logit model. NN outperformed multiple discriminant analysis.

(1991)

Dutta and Shekhar

(1988)
Remus and Hill

(1990)

transportation company financial performance bond rating production scheduling

4-10-1 did better than NN

random effects model but worse than fixed effects model. Ns outperformed multiple regression.

(1989)

Koster et al.

(1990)

1-1,1-12-1, 1-12-6-1
Ns did as well but not better than linear decision rules. Composite Ns did better than classical decision models.

Roy and Cosset

8-3-1 was comparable NN

to logistic regression NN model but 20-5-1 performed best. NN predicted likelihood of bankruptcy better than discriminant analysis. NN with 2 hidden layers performed best among the NN configurations. NN performed better than multiple discriminant analysis model.

(1990)

Remus and Hill

(1991)

production scheduling

Odom and Sharda

(1990)
Raghupthi et al.

bankruptcy prediction bankruptcy data on 102 companies stock price data on 98 companies

(1991)
Yoon and Swales (1990)

498