Week 2:

Estimation in time

Week 2: Estimation in time series models series models

Key references

Jiti Gao

March 5, 2023

Department of Econometrics and Business Statistics

1/51
 Week 2:
Recall that for a time series {Xt }, we define the Estimation in time
series models
autocovariance function by
Key references

γ(h) = cov(Xt+h , Xt ) (2.1)

= E (Xt+h − E [Xt+h ])(Xt − E [Xt ])

and the autocorrelation function (ACF) by

γ(h)
ρ(h) = corr(Xt+h , Xt ) = . (2.2)
γ(0)

2/51
Example 2.1 (MA(1) model): If Zt ∼ WN(0, σ 2 ), then Week 2:
Estimation in time
{Xt } defined by series models

Xt = Zt + θ1 Zt−1 Key references

is an MA(1) process, where θ1 is an unknown parameter.

The ACFs are given by

θ1
ρ(1) = and
1 + θ12
ρ(k) = 0, k ≥ 2. (2.3)

Note that MA processes are commonly used for error terms

in regression models.

3/51
 3
2
Week 2:

1
Estimation in time

ts.sim
series models

0
−1
Key references
−2
−3

0 50 100 150 200

Time

Series ts.sim
1.0
0.8
0.6
ACF

0.4
0.2
0.0

0 5 10 15 20

Lag

Figure: Plot of {Xt } with θ1 = 0.8 and its acf

4/51
 Week 2:
Example 2.2 (AR(1) model): Consider an AR(1) model of Estimation in time
series models
the form
Key references

Xt = φ1 Xt−1 + Zt , Zt ∼ WN(0, σ 2 ),

where φ1 is called an unknown parameter.

The ACFs are given by

γ(k)
ρ(k) = = φk1 , k = 1, 2, · · · . (2.4)
γ(0)

5/51
 3
2
Week 2:

1
Estimation in time

ts.sim

0
series models

−1
−2
Key references
−3
−4

0 20 40 60 80 100

Time

Series ts.sim
1.0
0.8
0.6
0.4
ACF

0.2
0.0
−0.2

0 5 10 15 20

Lag

Figure: Plot of an AR(1)–{Xt } with φ1 = 0.7 and its acf

6/51
In practice, one needs to estimate the autocovariance
function and autocorrelation function. Week 2:
Estimation in time
For a given sequence of observations x1 , x2 , · · · , xN of {Xt }, series models

define: Key references

N
1 X
x̄ = xi , (2.5)
N
i=1
N−k
1 X
ck = (xt − x̄)(xt+k − x̄), (2.6)
N −k
t=1
PN−k
ck N t=1 (xt − x̄)(xt+k − x̄)
rk = = PN
c0 N −k t=1 (xt − x̄)
2
PN−k
t=1 (xt − x̄)(xt+k − x̄)
≈ PN (2.7)
2
t=1 (xt − x̄)

k
when N is very small.
7/51
 Week 2:
Estimation in time
series models

Thus, ρ(k) can be estimated by rk . As N is large, Key references

rk ≈ ρ(k) for each given k ≥ 1. (2.8)

The calculation of the sample ACF of a time series data is

done using the function acf in R.

8/51
Method of Moments

The estimated ACFs can be used to estimate unknown Week 2:

Estimation in time
parameters in either MA(1) or AR(1) by solving series models

θ1 Key references
r1 = for the MA(1) case, (2.9)
1 + θ12
r1 = φ1 for the AR(1) case. (2.10)

Estimators of θ1 and φ1 , denoted by θb1 and φb1 , can then be

solved respectively from

r1 θb12 − θb1 + r1 = 0, (2.11)

φb1 = r1 . (2.12)

Such an estimation method is called the Method of

Moments. 9/51
Ordinary Least Squares Method Week 2:
Estimation in time
series models
To introduce the OLS estimation method, we have a look at
Key references
an AR(1) model of the form

Xt = φ1 Xt−1 + Zt . (2.13)

The idea of the OLS estimation method is to minimize

N
1 X
(Xt − φ1 Xt−1 )2 (2.14)
N
t=1

over φ1 .

10/51
 Week 2:
Estimation in time
This implies that the OLS estimator, φb1 , of φ1 is given by series models

PN Key references
t=1 Xt Xt−1
φ1 = P
b
N
. (2.15)
2
t=1 Xt−1

We then estimate σ 2 = E [Zt2 ] by

N
2 1 X 2
σ
b = Xt − φe1 Xt−1 . (2.16)
N
t=1

11/51
 Week 2:
Estimation in time
Note that series models
PN
t=1 (φ1 Xt−1 + Zt ) Xt−1 Key references
φb1 = PN 2
t=1 Xt−1
PN
Zt Xt−1
= φ1 + Pt=1 N
. (2.17)
2
t=1 Xt−1

Thus, it can be shown that

φb1 ≈ φ1 b2 ≈ σ 2 as N is large.
and σ (2.18)

12/51
 Week 2:
Estimation in time
series models

Example 2.3: In R, the OLS estimator can be calculated by Key references

using the following code function ar.ols:

ts.ols< − ar.ols(series, order.max=1)

Generate an AR(1) model of the form: Xt = 0.7Xt−1 + Zt

and then estimate φ1 = 0.7 by φb1 using the following code:

13/51
> ar.sim<-arima.sim(list(order=c(1,0,0),ar=0.7),
+ n=500) Week 2:
Estimation in time
> ts.ols<-ar.ols(ar.sim, order.max=1) series models

> ts.ols Key references

Call:
ar.ols(ar.sim, order.max = 1)
Coefficients:
1
0.718

Order selected 1, and $\sigma^2$ estimated as 0.99.

13/51
 Week 2:
Estimation in time
series models
We now extend the above discussion to an AR(p) model of
Key references

the form

Xt = φ1 Xt−1 + · · · + φp Xt−p + Zt , (2.19)

where Zt ∼ WN(0, σ 2 ), and φ1 , · · · , φp and σ 2 are unknown

parameters.

14/51
 Week 2:
Estimation in time
series models
The OLS estimation method is to minimize
Key references
N
1 X
(Xt − φ1 Xt−1 − · · · − φp Xt−p )2 (2.20)
N
t=1

over (φ1 , · · · , φp ).
Similarly to the AR(1) case, the OLS estimators,
(φb1 , · · · , φbp ), of (φ1 , · · · , φp ) can be explicitly expressed.

15/51
 Week 2:
Estimation in time
Meanwhile, the variance σ 2 = E [Zt2 ] can be estimated by series models

N Key references
2 1 X 2
σ
b = Xt − φb1 Xt−1 − · · · − φbp Xt−p . (2.21)
N
t=1

In theory, as N is large enough

φbi ≈ φi b2 ≈ σ 2
and σ (2.22)

for all 1 ≤ i ≤ p.

16/51
 Week 2:
Estimation in time
series models
Example 2.4: In R, the OLS estimator can be calculated by
Key references

using the following code function ar.ols:

ts.ols< − ar.ols(series, order.max=2)

Generate an AR(2) model of the form:

Xt = 0.7Xt−1 − 0.5Xt−2 + Zt and then use the following
code:

17/51
> ar.sim<-arima.sim(list(order=c(2,0,0),
+ ar=c(0.7, -0.5)), n=200) Week 2:
Estimation in time
> ts.ols<-ar.ols(ar.sim, order.max=2) series models

> ts.ols Key references

Call:
ar.ols(x = ar.sim, order.max = 2)
Coefficients:
1 2
0.6709 -0.5413

Intercept: -0.007082.
Order selected 2, and $\sigma^2$ estimated as 1.065.

17/51
 Week 2:
Estimation in time
series models
As shown above, both φb1 = 0.6709 and φb2 = −0.5403 are
Key references
close to φ1 = 0.7 and φ2 = −0.5, respectively.

b2 = 1.065 is very close to σ 2 = 1.

Meanwhile, σ

In the general real data application, we can always use the

following function in R:
> ts.ols < −ar.ols(realdata, order.max=p)

18/51
 Week 2:
Estimation in time
Yule–Walker estimation method series models

Key references
Consider an AR(p) model of the form,

Xt − φ1 Xt−1 − φ2 Xt−2 − · · · − φp Xt−p = Zt . (2.23)

Our aim is to find estimators of the coefficient vector

φ = (φ1 , . . . , φp ) and the white noise variance σ 2 based on
the observations X1 , . . . , XN .

19/51
 Week 2:
Estimation in time
series models
Multiplying each side of (2.23) by Xt−j , j = 0, 1, . . . , p,
Key references

Xt Xt − φ1 Xt Xt−1 − · · · − φp Xt Xt−p = Xt Zt ,

Xt Xt−1 − φ1 Xt−1 Xt−1 − · · · − φp Xt−1 Xt−p = Xt−1 Zt ,

.. .
. = ..,

Xt Xt−p − φ1 Xt−1 Xt−p − · · · − φp Xt−p Xt−p = Xt−p Zt ,

20/51
 Week 2:
Estimation in time
series models
We then have for j = 0, 1, . . . , p,
Key references

E [Xt Xt ] − φ1 E [Xt Xt−1 ] − · · · − φp E [Xt Xt−p ] = E [Xt Zt ],

E [Xt Xt−1 ] − φ1 E [Xt−1 Xt−1 ] − · · · − φp E [Xt−1 Xt−p ] = E [Xt−1 Zt ],
.. ..
. = .,
E [Xt Xt−p ] − φ1 E [Xt−1 Xt−p ] − · · · − φp E [Xt−p Xt−p ] = E [Xt−p Zt ].

21/51
 Week 2:
Estimation in time
Using the following results: γ(j) = E [Xt Xt−j ], series models

Xt = ∞
P
j=0 ψj Zt−j , Key references

∞
X
E [Xt Zt ] = ψj E [Zt Zt−j ] = E [Zt2 ] = σ 2 ,
j=0
X∞
E [Xt−k Zt ] = ψj E [Zt Zt−k−j ] = 0 (2.24)
j=0

for all k = 1, · · · , p, we finally have

22/51
 Week 2:
Estimation in time
series models

Key references

γ(0) − φ1 γ(1) − · · · − φp γ(p) = σ 2 ,

γ(1) − φ1 γ(0) − · · · − φp γ(p − 1) = 0,

.. ..
. = .,

γ(p) − φ1 γ(p − 1) − · · · − φp γ(0) = 0. (2.25)

23/51
Example 2.5. Consider the case where p = 2. We have that Week 2:
Estimation in time
series models
2
γ(0) − [φ1 γ(1) + φ2 γ(2)] = σ , Key references

γ(1) − φ1 γ(0) − φ2 γ(1) = 0,

γ(2) − φ1 γ(1) − φ2 γ(0) = 0. (2.26)

This implies that

1 − ρ(2) ρ(2) − ρ(1)2

φ1 = ρ(1) , φ 2 = ,
1 − ρ(1)2 1 − ρ(1)2
σ 2 = γ(0) − [φ1 γ(1) + φ2 γ(2)]. (2.27)

24/51
 Week 2:
Estimation in time
As in the Method of Moments, we replace γ(k) by ck and series models

ρ(k) by rk , respectively to define the estimators, φb1 , φb2 and Key references

b2 ,
σ of φ1 , φ2 and σ2 by the following estimation equations:

1 − r2
φb1 = r1 ,
1 − r12
r2 − r12
φb2 = ,
1 − r12
b2
σ = c0 − [φb1 c1 + φb2 c2 ]. (2.28)

25/51
 Week 2:
Estimation in time
series models
Example 2.6: In R, the Yule–Walker estimation method can
Key references
be implemented by using the following code function ar.yw:
ts.yw< − ar.yw(series, order.max=2)

Generate an AR(2) model of the form:

Xt = 0.7Xt−1 − 0.5Xt−2 + Zt and then estimate φ1 = 0.7 by
(φb1 , φb2 ) using the following code:

26/51
> ar.sim<-arima.sim(list(order=c(2,0,0),
+ ar=c(0.7, -0.5)), n=500)
> ts.yw<-ar.yw(ar.sim, order.max=2) Week 2:
Estimation in time
> ts.yw series models

Key references

Call:
ar.yw.default(x = ar.sim, order.max = 2)

Coefficients:
1 2
0.7502 -0.4607

Order selected 2, and $\sigma^2$ estimated as 1.094.

27/51
 Week 2:
Estimation in time
series models
Maximum likelihood (ML) estimation method
Key references

In addition to the Method of Moments, the OLS method

and the Yule–Walker estimation method for the AR case,
there is a general estimation method: the maximum
likelihood estimation method, which is applicable to the
general ARIMA structure.

27/51
 Week 2:
Consider an AR(1) model of the form Estimation in time
series models

Xt = φ1 Xt−1 + Zt , (2.29) Key references

where Zt ∼ WN(0, σ 2 ).
Let f (·, · · · , ·) be the joint density function of (Z1 , · · · , ZN ),
where Zt = Xt − φ1 Xt−1 . Define the likelihood function by

L(φ1 , σ; X1 , · · · , XN ) = f (Z1 , · · · , ZN )

= f (X1 − φ1 X0 , · · · , XN − φ1 XN−1 ).

28/51
 Week 2:
Estimation in time
The maximum likelihood estimation method is to choose series models

φb1 = φb1mle such that Key references

b2 ; X1 , · · · , XN )
L(φb1 , σ

= max L(φ1 , σ; X1 , · · · , XN ).
over all φ1 , σ 2

Both φ1 and σ 2 can then estimated by φb1mle and σ 2 ,

bmle
respectively.

29/51
 Week 2:
Example 2.7: Let {Zt } be a sequence of independent and Estimation in time
series models
normally distributed Zt ∼ N(0, σ 2 ). Then
Key references

N
Y N
Y
f (Z1 , · · · , ZN ) = f (Zt ) = f (Xt − φ1 Xt−1 )
t=1 t=1
N  (X −φ X )2

Y 1 − t 1 2t−1
= √ e 2σ

t=1
2πσ
 N PN (X −φ X )2
1 t=1 t 1 t−1
= √ e− 2σ 2 .
2πσ

30/51
 Week 2:
Estimation in time
series models

Thus, in order to maximize L(φ1 ; X1 , · · · , XN ) over φ1 , it Key references

suffices to minimize
N
X
(Xt − φ1 Xt−1 )2 over φ1 , (2.30)
t=1

which reduces exactly to the OLS estimation method.

31/51
 Week 2:
Estimation in time
series models

Key references

This shows that the ML estimation method is identical to

the OLS estimation method when {Zt } is a sequence of
independent and normally distributed Zt ∼ N(0, σ 2 ).

32/51
 Week 2:
Estimation in time
series models

Example 2.8: In R, the ML estimation method can be Key references

implemented by using the following code function arima:

ts.arima< − arima(series, order=c(p, d, q))

Consider an ARIMA(1,1,1) with, p = d = q = 1, φ1 = 0.7,

θ1 = 0.5 and n = 500.

33/51
simulated data
> ts2.sim<-arima.sim(list(order=c(1,1,1), Week 2:
Estimation in time
+ ar=0.7, ma=0.5), n=500) series models

> ts2.arima<-arima(ts2.sim, order=c(1,1,1)) Key references

> ts2.arima
Call:
arima(x = ts2.sim, order = c(1, 1, 1))
Coefficients:
ar1 ma1
0.6820 0.4411
$\sigma^2$ estimated as 0.9606:
log likelihood $= -700.1$.

33/51
 Week 2:
Estimation in ARCH and GARCH models Estimation in time
series models
Let {Zt2 } satisfy an ARCH(p) model of the form
Key references

Zt2 = α0 + α1 Zt−1
2 2
+ · · · + αp Zt−p + ut , (2.31)

where ut ∼ WN(0, σ 2 ).
Let Yt = Zt2 − E [Zt2 ]. Then {Yt } follows an AR(p) model of
the form

Yt = α1 Yt−1 + · · · + αp Yt−p + ut . (2.32)

34/51
 1 PN Week 2:
We estimate µ = E [Zt2 ] by µ
b= N
2
t=1 Zt and then Estimation in time
series models
estimate (α1 , · · · , αp ) by (b
α1 , · · · , α
bp ) using an existing
Key references
estimation method in R for an AR(p) model of the form

Ybt = α1 Ybt−1 + · · · + αp Ybt−p + ut , (2.33)

where Ybt = Zt2 − µ

b.
Finally, the parameter α0 is estimated by

α b (1 − α
b0 = µ b1 − · · · − α
bp ) . (2.34)

35/51
 Week 2:
Estimation in time
series models
Consider a GARCH(1,1) model of the form
Key references

Zt2 = α0 + (α1 + β1 )Zt−1

2
+ ut − β1 ut−1 , (2.35)

where ut ∼ WN(0, σ 2 ).
Let µ = E [Zt2 ] and Yt = Zt2 − µ. A simple derivation implies
α0
µ= 1−α1 −β1 .

36/51
 Week 2:
Estimation in time
The parameters α1 , β1 and α0 can then be estimated by α
b1 , series models

βb1 and α
b0 using a GARCH(1,1) model of the form Key references

Ybt = (α1 + β1 )Ybt−1 + ut − β1 ut−1 , (2.36)

which is an ARMA(1,1) with φ1 = α1 + β1 and θ1 = −β1 ,

and
 
α b 1−α
b0 = µ b1 − βb1 . (2.37)

37/51
 EuStockMarkets

80002000 4000 6000

Week 2:
Estimation in time

DAX
series models

Key references
SMI
5000
2000
CAC
3000
5000 1500
FTSE
3000

1992 1993 1994 1995 1996 1997 1998

Time

Figure: Plot of the Daily Closing Prices of Major European Stock

Indices: 1991–1998

38/51
 euro.dat1
Week 2:
Estimation in time

0.00
DAX
series models

0.00 −0.10 Key references

SMI
−0.08
0.06
CAC
0.00
−0.08
0.04
FTSE
−0.04 0.00

1992 1993 1994 1995 1996 1997 1998

Time

Figure: Plot of the log returns of the Daily Closing Prices

39/51
 Week 2:
Estimation in time
series models
Example: Consider the Daily Closing Prices of Major
Key references
European Stock Indices: 1991–1998. Let {Zt } be a
sequence the log returns and satisfy model (2.31).
The parameters α0 , α1 , · · · , αp can be estimated using the
following code function:

ts.arima< − arima(Ybt , order=c(p, 0, 0))

40/51
> euro.dat<-EuStockMarkets
> euro.dat1<-log(euro.dat) Week 2:
Estimation in time
> euro.dat2<-diff(euro.dat1) series models

> euro.dat3<-euro.dat2^2 - mean(euro.dat2^2) Key references

#DAX index
> euro.dat4<-euro.dat3[,1]
> euro.arima<-arima(euro.dat4, order=c(3,0,0))
> euro.arima

40/51
Call:
arima(x = euro.dat4, order = c(3, 0, 0)) Week 2:
Estimation in time
Coefficients: series models

ar1 ar2 ar3 intercept Key references

0.0574 0.1627 0.0508 0

sigma^2 estimated as 8.84 e-08:
log likelihood = 12458.51.
alpha_0 = 6.888995 e-05.

41/51
 Week 2:
Estimation in time
series models
Example 2.9: Consider the Daily Closing Prices of Major
Key references
European Stock Indices: 1991–1998. Let {Zt } be a
sequence the log returns and satisfy model (2.35).
The parameters α0 , α1 , β1 can be estimated using the
following code function:

ts.arima< − arima(Ybt , order=c(1, 0, 1))

41/51
> euro.dat<-EuStockMarkets
> euro.dat1<-log(euro.dat)
> euro.dat2<-diff(euro.dat1) Week 2:
Estimation in time
> euro.dat3<-euro.dat2^2 - mean(euro.dat2^2) series models

Key references

#DAX index
> euro.dat4<-euro.dat3[,1]
> euro.arima<-arima(euro.dat4, order=c(1,0,1))
> euro.arima

41/51
Call:
arima(x = euro.dat4, order = c(1, 0, 1))
Coefficients: Week 2:
Estimation in time
ar1 ma1 intercept series models

0.9150 -0.8386 0 Key references

$sigma^2$ estimated as 8.858e-08:

log likelihood = 12456.63.
$\beta_1 = 0.8386$,
$\alpha_1 = 0.9150 -0.8386=0.0754$,
$\alpha_0 = 8.031335e-06$.

42/51
Model Building: R provides some code functions for fitting Week 2:
Estimation in time
auto–regressive integrated moving–average (ARIMA) models series models

to univariate time series data. Key references

ARIMA models are useful for a wide variety of problems

including forecasting, quality control, seasonal adjustment,
spectral estimation, as well as providing a summary of the
data.

Box and Jenkins (1970) give a comprehensive account of

ARIMA modelling, and discussions of ARIMA models can be
found in many recent standard textbooks for time series.

42/51
Identifying and Fitting ARIMA Models. Box and Jenkins
(1970) give a paradigm for fitting ARIMA models, which is Week 2:
Estimation in time
to iterate through the following steps: series models

Key references
1. Model Identification: Determination of the ARIMA
model orders (p, d, q).

2. Estimation of Model Parameters: The unknown

parameters can be estimated through using the
maximum likelihood estimation method.

3. Diagnostics and Model Criterion: The residuals are used

to validate the model and to suggest potential
alternative models which may be better.

These steps are repeated until a satisfactory model is found.

43/51
 Week 2:
Estimation in time
series models

Model Identification. Initial model identification is done Key references

using the autocorrelation and partial autocorrelation

functions. These can be computed using R function acf.

An alternative procedure for selecting the model order is use

of a penalized log-likelihood measure.

44/51
 Week 2:

To introduce the idea, we start with an AR(p) model of the Estimation in time
series models

form
Key references

Xt = φ1 Xt−1 + · · · + φp Xt−p + Zt , t = 1, 2, · · · , N, (2.38)

where Zt ∼ WN(0, σ 2 ).
The estimated value of Xt and the one–step predictor is then
given by
Xbt = φb1 Xt−1 + · · · + φbp Xt−p . (2.39)

45/51
 Week 2:
Estimation in time

I An alternative view is adopted by Akaike. The problem series models

Key references
of model identification is then converted into an
optimization problem with respect to a specified
criterion over a pre-selected family of models.
I In what follows, we describe the approach developed by
Akaike (1974) in some detail. Limitation of both space
and our practical experience has indicated this choice.

46/51
 Week 2:
Estimation in time
series models
Consider an ARMA(p,q) process. The Akaike’s information
Key references

criterion (AIC) is defined as

σ 2 (p, q)) + 2(p + q),

AIC (p, q) = N log(b (2.40)

b2 (p, q) is the
where N is the number of observations, and σ
estimated error variance of σ 2 = E [Z12 ].

47/51
 Week 2:
Estimation in time
series models

An alternative that has better theoretical properties is the Key references

Bayesian information criterion (BIC) proposed by Schwarz

(1978),

σ 2 (p, q)) + log(N)(p + q).

BIC (p, q) = N log(b (2.41)

48/51
 Week 2:
Estimation in time
series models
The criteria are used in the following way. Upper bounds,
Key references
say P and Q, are set for the orders of φ(B) and θ(B), and
with P = {0, 1, . . . , Pmax } and Q = {0, 1, . . . , Qmax }, orders
pb and qb are selected such that, for example

AIC (b
p , qb) = min AIC (p, q). (2.42)
p∈P,q∈Q

49/51
 Week 2:
Estimation in time
Choice between AIC and BIC: series models

I AIC is more commonly used than BIC in practice; Key references

I In theory, BIC is asymptotically consistent in the sense

that pb ≈ p0 and qb ≈ q0 , but AIC is not asymptotically
consistent.

Computational Notes: R function arima fits ARIMA models

to univariate time series data through Gaussian maximum
likelihood.

50/51
A simulated example:
> ts.sim<-arima.sim(list(order=c(1,1,1), Week 2:
Estimation in time
+ ar=0.7, ma=0.5), n=500) series models

> ts.arima<-arima(ts.sim, order=c(1,1,1)) Key references

Call:
arima(x = ts.sim, order = c(1, 1, 1))
Coefficients:
ar1 ma1
0.6344 0.5180
s.e. 0.0398 0.0447
$\sigma^2$ estimated as 0.9749:
+ log likelihood = -703.81, aic = 1413.61.

50/51
Key references
Week 2:
Estimation in time
series models

Key references
Akaike, H. (1974). A new look at the statistical model
identification. IEEE Trans. Autom. Control. AC-19,
716–723.
Box, G. and Jenkins, G. (1970). Time Series Analysis,
Forecasting and Control. Holden–Day, San Francisco.
Schwarz, G. (1978). Estimating the dimension of a model.
Ann. Statist. 6, 461–464.

51/51