Customizing

Forecasting
Criteria
„how to win a forecasting
competition“
Results from the NN3-(2007) and
NN5-(2008)competitions
Forecasting Competitions
„ Not always well received in `academic´
world
„ Applied exercise: `tweaking´
„ `Real´ data enemy of standard procedures
„ High-risk (reputation)
„ Arbitrariness
„ Choice of time series (bias)
„ Choice of evaluation criteria
„ Significance of differences
„ Time-consuming (winner takes all)
Outcomes
„ Gain insight into current state-of-the-art
„ How well are theoretical expectations (about
performances) confirmed by empirics
„ Which kind of approach performs well in which context
„ M-competitions
„ Trivial models outperform traditional sophisticated ones
„ NN-competitions
„ Sophisticated customized approaches outperform trivial
ones.
The Traditional
Approach

An Illustrative Example
Statistics: One-Step-Ahead
Performances
„ Estimation:
„ Minimization of one-step-ahead mean-square
error
„ Identification
„ Information criteria
„ Diagnostics
„ One-step-ahead forecasting errors are iid
Example 1: ESI (EFN Report)

•Turning Point?
ARIMA(1,1,1)-Forecasting Model
(TRAMO/X-12-ARIMA)
Forecasts in TP April 1999
Forecasts one quarter after TP
Example 2 : KOF-Economic
Barometer (simulated example)
X t = 5*cos(π /18) + 0.5cos(π / 6) + ε t
Forecasting-Model and Diagnostics
(1 − B)(1 − B12 ) X t = (1 − 0.36 B)(1 − 0.99 B12 )ε t
σ ε = 1.16
Multi-step ahead Forecasts
0 months after TP1 of cycle
Multi-step ahead Forecasts
6 months after TP1 of cycle
Multi-step ahead Forecasts
1 year after TP1 of cycle
Multi-step ahead Forecasts 20 months
after TP1 and 0 months after TP2
Multi-step ahead Forecasts
3 months after TP 2
Multi-step ahead Forecasts
6 months after TP 2
Summary
„ One-step ahead performances are good
„ Multi-step ahead poor
„ Performance in turning-points?
„ Poor because integrated processes do not
assume `mean-reversion´
 NN3 Forecasting
Competition (2007)
A Prototype
Customization
 Receive updates:

„ www.neural-forecasting-competition.com
Objectives
Forecast a set of 111 monthly economic time series as accurately as possible (1-
18 step ahead), using methods from computational intelligence and a consistent
methodology. We hope to evaluate progress in modelling neural networks for
forecasting & to disseminate knowledge on “best practices”. The competition is
conducted for academic purposes and supported by a grant from SAS & the
International Institute of Forecasters (IIF).
Methods
The prediction competition is open to all methods of computational intelligence,
incl. feed-forward and recurrent neural networks, fuzzy predictors, evolutionary &
genetic algorithms, decision & regression tress, support vector regression, hybrid
approaches etc. used in financial forecasting, statistical prediction, time series
analysis
 Data
NN3 Fore ca sting Compe tition COMPLETE Dataset V2.0 www.neural-forecasting-competition.com

Instructions Informa tion a bout fie lds

Below you will find the data for 111 time series of the complete dataset (incl. the reduced dataset). Author information
The historical data (light green) is provided in vertical columns of different length.
Use each time series' training data to forecast the last 18 values of the test data (bright green). Information for each Time Series
Please fill in the results of your predictions on the next sheet "NN3 REDUCED Forecasts"
- Please fill in your out-of-sample predicitions on the test data on the next sheet nnnn Observation of Training Data
- Please fill in your in.sample predicitions on the training data on the next sheet.
Rename this excel file into "NN3_REDUCED_yourname.xls" ??? Test Data to be forecasted
Submit the complete file following the instrcutions on the webpage. GOOD LUCK!

Time Series #ID NN3-001 NN3-002 NN3-003 NN3-004 NN3-005 NN3-006 NN3-007 NN3-008 NN3-009 NN3-010 NN3-011 NN3-012 NN3-013 NN3-014 NN3-015 NN3-016 NN3-017
Start Year 1990 1990 1990 1990 1990 1990 1990 1990 1990 1990 1990 1990 1990 1990 1990 1990 1990
Start Month 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
Months per Year 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12
Dataset complete complete complete complete complete complete complete complete complete complete complete complete complete complete complete complete complete
5520 5080 6320 5500 4670 4540 7700 8300 5400 1880 2940 2300 5760 5520 7778 9520 8372
3940 3690 4770 3860 3150 4035 5700 7400 5200 1100 2720 2140 4190 3880 4960 5894 5820
4490 4260 5740 4880 3770 3725 6850 7800 4650 2160 3620 2760 4910 4840 6152 7400 7124
5030 3920 5360 4420 3380 3405 5200 9000 3950 1600 2680 1900 4870 4360 6210 7378 6218
5660 4290 4990 4900 3930 4195 5750 7200 4650 1900 3460 3140 4750 3800 6516 7112 7464
4790 3840 5330 4230 3350 3955 4500 6800 4500 1860 2880 2160 4930 4560 5904 7232 7200
5520 4830 5210 3970 3610 3840 5650 8300 4100 1780 3140 2060 4910 4560 5554 7702 7022
5560 4120 5470 4690 3740 3795 4900 5500 5450 2100 4000 3480 4750 5120 6298 7854 7198
5200 4530 4740 4190 3590 3495 4950 5400 5100 2940 4360 3340 5020 4320 5466 6324 6694
6670 4640 6280 4960 3730 4110 5650 7200 5250 2420 4220 3180 6380 5380 6482 8262 7812
5900 4510 6200 5590 3850 4005 7200 5000 5150 2660 4300 2800 5910 4860 6134 8248 8120
5280 4660 5640 5000 4010 3910 5350 6800 4200 1980 3020 2780 5880 4560 5446 7304 7192
6490 5340 7160 6030 4790 5355 8300 8400 6050 2520 4840 4180 7400 5820 7620 9692 9026
5560 3960 5530 4690 3330 3610 5700 7500 4700 2220 4080 2820 5300 4260 5628 7388 6780
5090 3680 4890 4090 2940 3395 5650 5500 4300 2920 4280 3500 5500 3720 5042 6290 6432
7090 4320 5660 5070 4020 4155 5350 5900 3750 2600 5540 3860 6290 4500 6062 7794 7140
6570 4420 6430 5050 4010 4345 5950 6100 4050 3300 5100 4040 6530 5200 6150 7586 7774
5890 4310 5270 4520 3450 3795 4750 6600 4250 2560 3820 2900 5560 4300 5750 7850 7784
6640 4410 6240 5070 4080 3960 6500 7200 4250 3160 4980 4400 6560 5460 5896 7634 7574
6360 4340 5220 4290 3590 3615 5900 6400 4350 2160 4720 3680 6270 4800 5752 7504 6468
5640 4010 4660 4400 3500 3755 5400 6300 3800 2160 4780 2960 5410 4500 5118 7094 7130
6630 4210 6110 5080 3680 3810 5300 5800 4300 2840 5160 4020 6220 4640 6196 8008 7468
5330 4330 5240 4180 3750 3570 6000 4600 3600 2400 4740 3360 5700 4600 6134 7514 7328
5540 4600 6680 5230 4370 4410 6850 6300 4750 3020 5580 3480 6930 5400 6694 8988 8080
7100 5140 6210 5200 4180 4345 7000 6600 5500 3580 6500 4820 6890 5220 6688 8674 7858
5410 4010 5320 3800 3350 3550 5150 5300 4600 2460 4360 3100 5390 3700 5578 7242 5472
5800 4440 6160 5010 3780 3685 6700 5300 4350 3060 5600 3200 6790 5180 6518 8520 7584
6110 4230 5230 4420 3490 3735 4850 6400 4000 2600 4820 3400 5810 4360 5658 6980 6048
6870 4160 5770 4810 3520 3905 6000 5300 4300 3740 6340 4320 6130 3920 5614 6596 7120
6560 4980 6100 4690 3880 4205 5150 5200 4200 3500 5880 4040 6440 4760 6504 8080 6906
6120 5130 6090 5390 4430 3770 5850 5600 3750 3000 6320 3600 6660 4600 5504 6778 7234
6540 4220 5810 4730 3420 3640 5750 5000 4050 2400 5540 3880 6470 4580 5838 7126 6682
5810 4520 5470 4770 3750 3995 5300 6800 4200 2440 4440 3140 5860 4300 5970 8284 7900
6300 4390 5780 4690 4010 3825 6100 4900 4300 2540 4800 3340 5910 4920 6152 7786 6582
6270 4420 5570 4450 3940 3885 4400 8000 4150 2420 5320 3540 6240 3940 6030 7702 7976
6900 5620 6660 5400 4330 4265 6850 5100 5050 2800 6000 4800 7160 5640 6730 9572 7358
6310 4510 6030 5590 4160 4265 5450 6500 4600 4360 7040 5400 6450 4800 6366 7466 7494
5330 4520 5600 4360 3520 3365 6450 8100 3950 2880 5320 3680 5340 4120 5864 6922 6406
5700 5120 6870 5370 3710 4515 6950 6700 5000 3340 7240 5860 6750 4940 7614 9002 8642
6680 4090 5190 4660 3650 3680 3900 6100 3350 2780 5840 4000 6420 4540 5674 7678 6854
5510 4000 5060 4450 3460 3405 4450 5000 4000 3120 5660 4380 5670 4100 4928 6358 5966
6690 4740 6690 4980 3770 4080 5350 6400 5150 2420 5180 4080 6490 4940 6424 8728 7560
5870 4360 5560 4590 4140 3575 5050 4500 3700 3260 6600 4780 5690 4660 5750 7140 7168
7140 3920 5900 4580 3470 3335 4250 4400 2950 3080 6480 4180 6220 4180 5844 6996 6196
6680 4610 5610 4290 3730 3795 5150 6400 4450 2860 5860 4060 6060 3960 5854 7316 7302
6520 4590 5420 4840 3590 3545 4800 4500 3600 3080 6620 4620 6220 4380 6080 7684 6780
6020 4140 5300 5100 3970 3900 4200 5200 4000 2680 5560 4320 6610 4480 5876 7614 7132
6190 5040 7040 6170 4220 4290 6350 6800 5050 3260 5980 4800 7750 5860 6848 8730 7446
6690 5860 7840 5990 5050 4640 5900 5300 4850 3380 6740 4760 8580 4760 7502 9310 8922
6330 4240 6640 4950 4090 4010 5350 5200 3900 3480 5980 4780 6430 4440 6734 7218 7076
7620 4230 6090 5310 3630 4025 4800 6500 4300 4680 8940 6260 6980 5020 6440 7800 7630
??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ???
??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ???
??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ???
??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ???
??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ??? ???
Data/Criterion
„ Subset of M3-Competition
„ Length between 50 and 110 observations
„ Economic monthly data
„ Finance
„ Macroeconomic data
„ With/without season
„ MAPE on 1-18 step-ahead forecasts
Model
Component- (State-Space-) Model
ξt = Fξt −1 +ν t
X t = Tt + St + I t
X t = H ξt + I t ⎛ν t1 ⎞
⎛ Tt ⎞ ⎜ ⎟
⎜
ΔT
⎟ ⎜ν t 2 ⎟
⎜ t ⎟ ⎜ν ⎟
ξ t = ⎜ St ⎟ , H=(1,1,1,0,0,0,0,0,0,0,0,0,0,0,0) , ν t = ⎜ t 3 ⎟
⎜ ⎟ ⎜0 ⎟
⎜ ... ⎟ ⎜ ... ⎟
⎜S ⎟ ⎜⎜ ⎟⎟
⎝ t −13 ⎠
⎝0 ⎠
⎛1,1,0, 0 ,0,0,0,0,0,0,0,0,0, 0 , 0 ⎞
⎜ ⎟
⎜ 0,1,0, 0 ,0,0,0,0,0,0,0,0,0, 0 , 0 ⎟
F = ⎜ 0,0,a1 ,(1-IS=T )*a 2 ,0,0,0,0,0,0,0,0,0,IS=T *a12 ,-IS=T *a1a12 ⎟
⎜ ⎟
⎜ 0,0,0 , 1 ,0, ... , 0 ⎟
⎜ 0,0,0 , ... , 1 ⎟
⎝ ⎠
 Modifications

Optimization Criterion
Experience ↔ 6 Modifications
„ Past Competitions
„ Fit model parm. according to relevant criterion #3
„ Performance dependent on Forecasting horizon #4
„ Model-misspecification
„ Combination of forecasts often improves over individual
forecasts #5
„ Own experience
„ Out-of sample performance #1
„ Truncation (robustification vs. outlier analysis) #2
„ Speed of trend-slope estimate #6
 Modifications: #1, #2, #3: out-of-
sample, `robustification’, Criterion
Xˆ t = H ξt|t −1 ←⎯
→ Xˆ t = H ξt|t
⎧⎪X t -X ˆ , ˆ |<2.5*Median(|I |,...,|I |)/0.6745
|X t -X
→ I t = X t − Xˆ t
t t t-10 t
It = ⎨ ←⎯
⎪⎩sign(X t -X ˆ )*2.5*Median(|I |,...,|I |)/0.6745 , otherwise
t t-10 t

ξt|t = ξt|t −1 + Pt|t −1 H ( H ' Pt|t −1 H + R) −1 I t

ξt +1|t = Fξt|t
Pt|t = Pt|t −1 − Pt|t −1 H ( H ' Pt|t −1 H + R) −1 Pt|t −1
Pt +1|t = FPt|t F '+ Q
⎛ T-forestep T+1-t forestep ⎞ ⎧⎪ T ⎛ |I t |2 ⎞ ⎫⎪
Criterion= ⎜ ∑ α |I t | ⎟ + K ⎨∑ ⎜ log(σ t )+ 2 ⎟ ⎬
2

⎝ t=1 ⎠ ⎩⎪ t=1 ⎝ σ t ⎠ ⎭⎪

• User Preferences, data-charateristics, flexibility (overparameterization)

Modifications #4 and #5
„ Optimize parameters depending on
forecasting horizon h=1,…,18
„ Combine Forecasts from h-step ahead
models
Modification #6: `Speed’ and
`Reliability’ through TP-Filter
„ A fast and reliable TP-filter is computed
⎧ 2π [T / 2]
⎪⎪ T ∑ k =0 W (ω ) 2
| Γ (ω ) − ˆ (ω ) |2 I (ω )
A
k k k NX k

„ DFA: min Γˆ ⎨
⎪+λ 2π ∑ [T / 2] 2W (ω ) 2 A(ω ) Aˆ (ω )(1 − cos(Φ ˆ (ω )))I (ω )
⎪⎩ T k =0 k k k k NX k

„ Correct ΔTt
„ Important for multi-step ahead performance
„ Latest book available on institute’s Website
Comparison with
Traditional BSM
Basic Structural Model
First 10 Series of NN3
Series 3
Series 4
Series 5
Series 6
Series 7
Series 8
Series 9
Series 10
 NN5 (2008)
Modifications of the Prototype
New Customization
„ Cash money demand (withdrawals at cash-machines ) represents a non-
stationary, heteroscedastic process. The time series features, regular
trend-seasonal and irregular structural components of the data as well as
causal forces impacting on the data generating process were already
indicated in section 1. The data will consist of a data set of 11 and 111
empirical time series of 2 years of daily data, provided by an unknown
source. The data has not been used in previous competitions to prevent
overfitting to the domain or dataset. All data is linearly scaled to ensure
anonymity of the time series.
„ The competition design and dataset adhere to previously identified
requirements from major forecasting competitions in the statistics and
econometrics domain (Fildes et al., 1998; Makridakis and Hibon, 2000) as
well as set out through the International Journal of Forecasting and build
upon experience from the preceding NN3-competition in the CI-domain in
order to derive valid and reliable competition results
„ SMAPE-Criterion
„ 111 daily data
„ Plus Focus on 11 “difficult” time series
„ Length: 750
„ Forecasting horizon: 1- to 56-step ahead
„ Eastern, Bank-holidays in forecasting horizon
„ More than 50 Competitors
„ Forecast-Pro (best commercial package M3)
„ Autobox (ARIMA-based high-performer)
„ Two expert-contenders (strong domain-knowledge)
„ large forecasting projects with the industry in USA and UK
„ Newest developments in artificial intelligence
Data-Characteristics and Salient
Features
„ NA‘s (particularly in periods corresponding to
forecasting horizon)
„ Zeroes (`outliers´)
„ Dynamics:
„ Multiple Seasonalities
„ Week, month, year
„ holidays, christmas, eastern, bank-holidays
„ Seasonality may appear like trend within year
„ Irregular (SMAPE of winner 19%)
„ Trend
Seasonality
„ „Short-term“ variable
weekday-effects
„ „Long-term“ year-
effect
„ Trend
„ Holidays
„ Eastern shifts!
„ 1-B364 Filter ‚bad‘ idea
Ideas
„ Long-term seasonality (trend and holidays)
„ Smooth-out irregular and week-day effect
„ Lowess with optimized bandwith
„ Easter-effects
„ Smooth-out other effects
„ Take mean of two years
„ Shift mean-effect according to year 1998 of
forecasting horizon
Long-Term Seasonality (Bandwith
Optimized from State-Space Model)
mplot[, 1]

15
10
5
0

0 100 200 300

1:length(mplot[, 1])
Easter
14
12
10
8
6
4

0 200 400 600

Time
Forecast: Long-Term Season Lagged 1-
Year without Eastern (Level not Optimal!)
14
12
10
8
6
4

0 200 400 600 800

Time
Forecast: Lagged Long-Term Season with
Shifted Mean-Eastern (Level not Optimal!)
14
12
10
8
6

0 20 40 60 80

Time
 Model
State Space
Link to Lowess (Bandwith)
Optimal Level
Short-Term (Weekday) Adaptive Season
Component- (State-Space-) Model
X t = Tt + St + αΔLt + Eastert + Bankt + I t

ξt = Fξt −1 +ν t
X t = H ξt + αΔLt + Eastert + Bankt + I t
⎛ Tt ⎞
⎜ ⎟
⎜ C11t ⎟ ⎛ν t1 ⎞
⎜ C12t ⎟ ⎜ ⎟
⎜ ⎟ ν
ξ t = ⎜ C22t ⎟ , H=(1,1,1,1,1,1,1) , ν t = ⎜ t 2 ⎟
⎜ ... ⎟
⎜C ⎟ ⎜ ⎟
⎜ 22t ⎟ ⎝ν t 7 ⎠
⎜ C32t ⎟
⎜ ⎟
⎝ C32t ⎠
⎛1, 0, 0, 0, 0, 0, 0 ⎞
⎜ ⎟
⎜ 0, ρ1cos(2π/7),ρ1sin(2π/7), 0, 0, 0, 0 ⎟
⎜ 0, ρ1sin(2π/7),-ρ1cos(2π/7), 0, 0, 0, 0 ⎟
⎜ ⎟
F = ⎜ 0, 0, 0, ρ 2 cos(4π/7),ρ 2sin(4π/7), 0, 0 ⎟
⎜ 0, 0, 0, ρ 2sin(4π/7),-ρ 2 cos(4π/7), 0, 0 ⎟
⎜ ⎟
⎜ 0, 0, 0, 0, 0, ρ3cos(6π/7),ρ3sin(6π/7) ⎟
⎜ 0, 0, 0, 0, 0, ρ3sin(6π/7),-ρ3cos(6π/7) ⎟⎠
⎝
Customization
„ Same modifications as for NN3 except that
fast TP-filter is replaced by 1-year lagged
long-term seasonality
Salient Features
„ Level :
„ Endogeneous by Tt
(customized) Kalman-Filter
„ Long-Term Seasonality:
„ Exogeneous by lowess- αΔLt
smoother
„ Bandwith optimized
„ Short-Term Seasonality St = ∑ Cijt
„ Endogeneous by Kalman- ij
Filter
Short-Term Seasonality (Weekday)
Adaptivity

4
apply(zyklenis[, c(3:8)], 1, sum)

2
0
-2
-4

0 200 400 600

Time
„Long-Term“ Seasonality and (Kalman-)
Level: Forecasts + Shifted Easter-Effect
15
c(x[anf:len], rep(NA, anzfor))

10
5
0

0 100 200 300 400

1:(len - anf + 1 + anzfor)

Final Forecasts: Level + Long- + Short-
term Seasonalities + Shifted Easter-Effect
+ Bank-Holidays
15
10
5
0

0 50 100 150

Time
Summary
Findings
„ Since the M-competitions there is a strong belief
among forecasters that
„ Trivial models/approaches perform better than
sophisticated ones
„ Statistics/Econometrics are `useless´
„ Increasing role of Judgmental Forecasts (combination
with simple models)
„ NN-competitions rehabilitate to some extent
`serious´ statistical/econometric work
„ Seriousness: to account for data characteristics and user
preferences alleviates model-misspecification and
overfitting
Contact
„ E-Mail
marc.wildi@zhaw.ch
„ Institute
http://www.idp.zhwin.ch
„ Forecasting:
http://www.idp.zhaw.ch/de/engineering/idp/forschung/finance-
risk-management-and-econometrics/signal-extraction-and-
forecasting/forecasting.html
„ Real-time Signal-Extraction:
http://www.idp.zhaw.ch/de/engineering/idp/forschung/finance-
risk-management-and-econometrics/signal-extraction-and-
forecasting/signal-extraction.html
„ Books, articles, projects, data, software (R-packages)
„ R-Package `signalextraction´ on CRAN
Design of Future
Forecasting
Competitions
Practically Relevant Dynamic Setting
„ Traditional competitions: Static Framework
„ `Real-time‘ time series
„ Data subject to revisions
„ Each month a new observation
„ Dynamic revisions
„ One- and multi-step ahead (in real-time)
„ Outliers/shifts (in real-time)
„ Turning-points (in real-time)