Professional Documents
Culture Documents
AR, MA and ARMA Models: Stationarity ACF Ljung-Box Test White Noise AR Models Example Pacf Aic/Bic Forecasting MA Models
AR, MA and ARMA Models: Stationarity ACF Ljung-Box Test White Noise AR Models Example Pacf Aic/Bic Forecasting MA Models
Stationarity 1 Stationarity
ACF
Ljung-Box
2 ACF
test
White noise
3 Ljung-Box test
AR models
4 White noise
Example
PACF 5 AR models
AIC/BIC
6 Example
Forecasting
MA models 7 PACF
Summary
8 AIC/BIC
9 Forecasting
10 MA models
11 Summary
Linear Time Series Analysis
Stationarity
ACF
and Its Applications1
Ljung-Box
test
For basic concepts of linear time series analysis see
White noise
• Box, Jenkins, and Reinsel (1994, Chapters 2-3), and
AR models
AIC/BIC
1
Tsay (2010), Chapter 2.
Stationarity
ACF
White noise
(a) simple autoregressive models,
AR models
(b) simple moving-average models,
Example
PACF
(b) mixed autoregressive moving-average models,
AIC/BIC
(c) seasonal models,
Forecasting
(d) unit-root nonstationarity,
MA models (e) regression models with time series errors, and
Summary (f) fractionally differenced models for long-range
dependence.
Strict stationarity
Stationarity
ACF
Ljung-Box
test The foundation of time series analysis is stationarity.
White noise
MA models
The joint distribution of (rt1 , . . . , rtk ) is invariant under time
Summary
shift.
ACF
A time series {rt } is weakly stationary if both the mean of rt
Ljung-Box
and the covariance between rt and rt−l are time invariant,
test
where l is an arbitrary integer.
White noise
AR models
MA models
In practice, suppose that we have observed T data points
Summary
{rt |t = 1, . . . , T }. The weak stationarity implies that the
time plot of the data would show that the T values fluctuate
with constant variation around a fixed level.
Example
MA models
where t1 = t + l.
Autocorrelation function
Stationarity
ACF
PACF
where the property V ar(rt ) = V ar(rt−l ) for a weakly
AIC/BIC
stationary series is used.
Forecasting
MA models
ACF
Box and Pierce (1970) propose the Portmanteau statistic
Ljung-Box
test
m
White noise X
AR models
Q∗ (m) = T ρ̂2l
l=1
Example
PACF
as a test statistic for the null hypothesis
AIC/BIC
Forecasting H0 : ρ1 = · · · = ρm = 0
MA models
ACF
Ljung-Box
test
White noise Ljung and Box (1978) modify the Q∗ (m) statistic as below
AR models to increase the power of the test in finite samples,
Example
m
PACF X ρ̂2l
Q(m) = T (T + 2) .
AIC/BIC T −l
l=1
Forecasting
MA models
Summary
> Box.test(sibm2,lag=30,type="Ljung")
Box-Ljung test
data: sibm2
X-squared = 182.12, df = 30, p-value < 2.2e-16
White noise
Stationarity
ACF
Ljung-Box
test
AIC/BIC
Summary
If rt is normally distributed with mean zero and variance σ 2 ,
the series is called a Gaussian white noise.
Linear Time Series
Stationarity
A time series rt is said to be linear if it can be written as
ACF
∞
X
Ljung-Box
test rt = µ + ψi at−i ,
White noise i=0
AR models
where µ is the mean of rt , ψ0 = 1, and {at } is white noise.
Example
PACF
Summary
Ljung-Box
test
γl = Cov(rt , rt−l )
! ∞
White noise ∞
X X
AR models = E ψi at−i ψj at−l−j
Example i=0 j=0
PACF
∞
X
AIC/BIC
= E ψi ψj at−i at−l−j
Forecasting
i,j=0
MA models
∞
X ∞
X
ψj+l ψj E(a2t−l−j ) = σa2
Summary
= ψj ψj+l ,
i=0 j=0
so P∞
γl ψψ
ρl = P∞i i+l2
= i=0
γ0 1 + i=1 ψi
AR(1)
Stationarity
AR models
rt − µ = φ1 (rt−1 − µ) + at
Example
PACF
where {at } is white noise. It is easy to see that
AIC/BIC
Forecasting ∞
X
MA models rt − µ = φi1 at−i ,
Summary i=0
and
σa2
V (rt ) = ,
1 − φ21
provided that φ21 < 1. In other words, the weak stationarity
of an AR(1) model implies that |φ1 | < 1.
Using φ0 = (1 − φ1 )µ, one can rewrite a stationary AR(1)
Stationarity model as
ACF rt = φ0 + φ1 rt−1 + at ,
Ljung-Box
test such that φ1 measures the persistence of the dynamic
White noise dependence of an AR(1) time series.
AR models
Example
PACF
The ACF of the AR(1) is
AIC/BIC
Forecasting
γl = φ1 γl−1 l > 0,
MA models
Summary
where γ0 = φ1 γ1 + σa2 and γl = γ−l .
Also,
ρl = φl1 ,
i.e., the ACF of a weakly stationary AR(1) series decays
exponentially with rate φ1 and starting value ρ0 = 1.
AR(2)
Stationarity
ACF
An AR(2) model assumes the form
Ljung-Box
test rt = φ0 + φ1 rt−1 + φ2 rt−2 + at ,
White noise
AR models where
φ0
Example E(rt ) = µ = ,
PACF 1 − φ1 − φ2
AIC/BIC provided that φ1 + φ2 6= 1.
Forecasting
MA models
and that
ρl = φ1 ρl−1 + φ2 ρl−2 , l ≥ 2,
with ρ1 = φ1 /(1 − φ2 ).
US real GNP
Stationarity
ACF
Ljung-Box
test
White noise
As an illustration, consider the quarterly growth rate of U.S.
AR models real gross national product (GNP), seasonally adjusted,
Example from the second quarter of 1947 to the first quarter of 1991.
PACF
AIC/BIC
Forecasting
Here we simply employ an AR(3) model for the data.
MA models
Denoting the growth rate by rt the fitted model is
Summary
rt = 0.0047 + 0.348rt−1 + 0.179rt−2 − 0.142rt−3 + at ,
Ljung-Box
rt − 0.348rt−1 − 0.179rt−2 + 0.142rt−3 = 0.0047 + at ,
test
AIC/BIC or
Forecasting (1 + 0.521B)(1 − 0.869B + 0.274B 2 ) = 0
MA models
Summary
Forecasting
Fact: If one uses a nonlinear model to separate U.S. economy into
MA models
“expansion” and “contraction” periods, the data show that the
Summary
average duration of contraction periods is about 3 quarters and
that of expansion periods is about 12 quarters.
PACF
The mean of a stationary series is
AIC/BIC
Forecasting φ0
E(rt ) =
MA models 1 − φ1 − · · · − φp
Summary
provided that the denominator is not zero.
ACF
Ljung-Box
test The PACF of a stationary time series is a function of its
White noise ACF and is a useful tool for determining the order p of an
AR models AR model. A simple, yet effective way to introduce PACF is
Example to consider the following AR models in consecutive orders:
PACF
The estimate φ̂i,i the ith equation is called the lag-i sample
PACF of rt .
Stationarity
ACF
Ljung-Box
test For a stationary Gaussian AR(p) model, it can be shown
White noise that the sample PACF has the following properties:
AR models
• φ̂p,p → φp as T → ∞.
Example
Summary
PACF cuts off at lag p.
AIC
Stationarity
ACF
The well-known Akaike information criterion (AIC) (Akaike,
Ljung-Box
1973) is defined as
test
White noise 2 2
AIC = − log(likelihood) + (number of parameters),
AR models
| T {z } |T {z }
Example
goodness of fit penalty function
PACF
Summary
2l
AIC(l) = log(σ̃l2 ) +
T
where σ̃l2 is the maximum-likelihood estimate of σa2 , which is
the variance of at and T is the sample size.
BIC
Stationarity
AR models
For a Gaussian AR(l) model, the criterion is
Example
PACF
l log(T )
AIC/BIC BIC(l) = log(σ̃l2 ) +
T
Forecasting
MA models
Summary
The penalty for each parameter used is 2 for AIC and log(T )
for BIC.
ACF
For the AR(p) model, suppose that we are at the time index
Ljung-Box h and are interested in forecasting rh+l where l ≥ 1.
test
White noise
The time index h is called the forecast origin and the
AR models
positive integer l is the forecast horizon.
Example
PACF
AIC/BIC
Let r̂h (l) be the forecast of rh+l using the minimum squared
Forecasting error loss function, i.e.
MA models
Summary
E{[rh+l − r̂h (l)]2 |Fh } ≤ min E[(rh+l − g)2 |Fh ],
g
ACF
Ljung-Box
test
It is easy to see that
White noise
AR models p
X
Example r̂h (1) = E(rh+1 |Fh ) = φ0 + φi rh+1−i ,
PACF i=1
AIC/BIC
Forecasting
and the associated forecast error is
MA models
eh (1) = rh+1 − r̂h (1) = ah+1 ,
Summary
and
V (eh (1)) = V (ah+1 ) = σa2 .
2-step-ahead forecast
Stationarity
ACF
Ljung-Box
test
White noise
Similarly,
AR models
Example
r̂h (2) = φ0 + φ1 r̂h (1) + φ2 rh + · · · + φp rh+2−p ,
PACF
AIC/BIC
with
Forecasting
eh (2) = ah+2 + φ1 ah+1
MA models
Summary and
V (eh (2)) = (1 + φ21 )σa2 .
Multistep-ahead forecast
Stationarity
ACF
Ljung-Box
test
PACF
rh+l = φ0 + φi rh+l−i + ah+l ,
i=1
AIC/BIC
Forecasting
and
p
MA models X
Summary r̂h (l) = φ0 + φi r̂h (l − i),
i=1
ACF
It can be shown that for a stationary AR(p) model,
Ljung-Box
test r̂h (l) → E(rt ) mboxas l → ∞,
White noise
AIC/BIC
log(0.5)
half-life = .
log(|φ1 |)
MA(1) model
Stationarity
ACF
Ljung-Box
test
There are several ways to introduce MA models.
White noise
AR models
Example
One approach is to treat the model as a simple extension of
PACF
white noise series.
AIC/BIC
Forecasting
AR models
infinite many parameters.
Example
PACF
One way to make the model practical is to assume that the
AIC/BIC
coefficients φi ’s satisfy some constraints so that they are
Forecasting
determined by a finite number of parameters.
MA models
Summary
ACF Obviously,
Ljung-Box
test rt + θ1 rt−1 + θ12 rt−2 + θ13 rt−3 + · · · = φ0 + at
θ12 rt−3 θ13 rt−4
White noise
θ1 (rt−1 + θ1 rt−2 + + + ···) = θ1 (φ0 + at−1 )
AR models
Example so
PACF
rt = φ0 (1 − θ1 ) + at − θ1 at−1 ,
AIC/BIC
Summary
Therefore, the model is called an MA model of order 1 or
M A(1) model for short.
MA(q)
Stationarity
ACF
The general form of an MA(q) model is
Ljung-Box q
test X
White noise
rt = c0 − θi at−i ,
AR models
i=1
Example or
PACF rt = c0 + (1 − θ1 B − · · · − θq B q )at ,
AIC/BIC
Forecasting
where q > 0.
MA models
Summary
Moving-average models are always weakly stationary
because they are finite linear combinations of a white noise
sequence for which the first two moments are time invariant.
E(rt ) = c0
V (rt ) = (1 + θ12 + θ22 + · · · + θq2 )σa2 .
ACF of an MA(1)
Stationarity
ACF
Ljung-Box
Assume that c0 = 0 for simplicity. Then,
test
Example
Summary
θ1
ρ0 = 1, ρ1 = − , ρl = 0, for l > 1.
1 + θ12
ACF of an MA(2)
Stationarity
Example
Here the ACF cuts off at lag 2.
PACF
AIC/BIC
Summary
For an MA(q) model, the lag-q ACF is not zero, but ρl = 0
for l > q.
ACF
White noise
estimate MA models. There are two approaches for
AR models
evaluating the likelihood function of an MA model.
Example
PACF
The first approach assumes that at = 0 for t ≤ 0, so
AIC/BIC
a1 = r1 − c0 , a2 = r2 − c0 + θ1 a1 , etc. This approach is
Forecasting
MA models
referred to as the conditional-likelihood method.
Summary
ACF
For the 1-step-ahead forecast of an MA(1) process, the
Ljung-Box model says
test
rh+1 = c0 + ah+1 − θ1 ah .
White noise
PACF
r̂h (1) = E(rh+1 |Fh ) = c0 − θ1 ah ,
AIC/BIC eh (1) = rh+1 − r̂h (1) = ah+1
Forecasting
MA models
with V [eh (1)] = σa2 .
Summary
Similarly,
ACF
Ljung-Box
test
MA models
r̂h (1) = c0 − θ1 ah − θ2 ah−1 ,
Summary r̂h (2) = c0 − θ2 ah ,
r̂h (l) = c0 , for l > 2.
Summary
Stationarity
ACF
AR models
• For MA models, ACF is useful in specifying the order
Example because ACF cuts off at lag q for an MA(q) series.
PACF • For AR models, PACF is useful in order determination
AIC/BIC because PACF cuts off at lag p for an AR(p) process.
Forecasting
• An MA series is always stationary, but for an AR series
MA models
Summary
to be stationary, all of its characteristic roots must be
less than 1 in modulus.