Unit 11

ACF for MA(q) Processes ACF for Causal ARMA(p,q) Processes Partial Autocorrelation Function Worked Examples
Unit 11: ARMA Autocorrelation and Partial

Autocorrelation Functions
Taylor R. Brown
Department of Statistics, University of Virginia
Fall 2020
1 / 37
Readings for Unit 11
Textbook chapter 3.3.
2 / 37
Last Unit
1 ARMA(p,q)
2 Condition for causality
3 Condition for invertibility
4 Condition for redundant parameters (shared roots)
3 / 37
This Unit
1 ACF for MA(q)

2 ACF for Causal ARMA(p,q)
3 Partial Autocorrelation Function (PACF)
4 / 37
Motivation
In this unit we will study the autocorrelation and partial

autocorrelation functions for ARMA processes.
5 / 37
MA(q) Process
Let’s start with an MA(q) process

q
X
xt = wt + θ1 wt−1 + θ2 wt−2 + · · · + θq wt−q = θj wt−j ,
j=0
where we have written θ0 = 1. Then

q
X
E (xt ) = θj E (wt−j ) = 0.
j=0
6 / 37
Autocovariance for MA(q)
The autocovariance function is

q
hX q
X i
γ(h) = cov (xt , xt+h ) = E θj wt−j θj 0 wt+h−j 0
j=0 j 0 =0
q X
X q
= θj θj 0 E (wt−j wt+h−j 0 ).
j=0 j 0 =0
7 / 37
Autocovariance for MA(q)
Recall that E (ws wt ) = σw2 if s = t and E (ws wt ) = 0 otherwise. So

we have
(
σw2 q−h
P
γ(h) = j=0 θj θj+h , 0 ≤ h ≤ q, (1)
0, h ≥ q + 1.
8 / 37
ACF for MA(q)
Recall that γ(h) = γ(−h), so we will only need the values for
h ≥ 0. Dividing γ(h) by γ(0) in (1), we obtain the autocorrelation
function (ACF) of an MA(q) model
 Pq−h
 j=0 θj θj+h
1+θ12 +···+θq2
, 0 ≤ h ≤ q,
ρ(h) = (2)
 0, h ≥ q + 1.
9 / 37
1 ACF for MA(q) Processes
2 ACF for Causal ARMA(p,q) Processes
3 Partial Autocorrelation Function
4 Worked Examples
10 / 37
ACF for Causal ARMA(p,q)
We have seen in (2), for MA(q) models, the ACF will be zero for
lags greater than q. Moreover, because θq 6= 0,
ρ(q) = θ0 θq /(1 + θ12 + · · · + θq2 ) 6= 0. Thus, the ACF provides
information about the order of the dependence for a MA model.
How about ARMA or AR models?
11 / 37
Causal ARMA(p,q)
Now we discuss causal ARMA(p, q) model
φ(B)xt = θ(B)wt ,
where the roots of φ(z) are outside the unit circle. Ths means for
|z| ≤ 1, |φ(z)| > 0, which means |φ−1 (z)| < ∞.
We have the MA(∞) representation

∞ ∞
X θ(z) X
xt = ψj wt−j , where ψ(z) = = ψj z j . (3)
φ(z)
j=0 j=0
12 / 37
Autocovariance for Causal ARMA(p,q)
It follows that E (xt ) = 0 and by (1), the autocovariance function

of xt is given by
∞
X
γ(h) = cov (xt , xt+h ) = σw2 ψj ψj+h , h ≥ 0.
j=0
13 / 37
Autocovariance for Causal ARMA(p,q)

We won’t provide an explicit formula for the ACF of an
ARMA(p,q), but we will derive the system of equations you’ll need
to solve to get it.
Let h ≥ 0, and θ0 = 1:
γX (h) = Cov(Xt+h , Xt )
Xp q
X
= Cov( φi Xt+h−i + θj Wt+h−j , Xt )
i=1 j=0
p
X q
X
= φi Cov(Xt+h−i , Xt ) + θj Cov (Wt+h−j , Xt )
i=1 j=0
p q ∞
!
X X X
= φi γX (h − i) + θj Cov Wt+h−j , ψk Wt−k
i=1 j=0 k=0
14 / 37
ACVF and ACF for Causal ARMA(p,q)
Let h ≥ 0, and θ0 = 1:
p q ∞
!
X X X
γX (h) = φi γX (h − i) + θj Cov Wt+h−j , ψk Wt−k
i=1 j=0 k=0
Xp Xq X ∞
2
= φi γX (h − i) + θj ψk σW 1(t + h − j = t − k)
i=1 j=0 k=0
p
X Xq
2
= φi γX (h − i) + σW θj ψj−h
i=1 j=h
0 ≤ j < ∞, k ≥ 0, and k = j − h. Dividing through by γ(0) will

give the equations for solving ρ(·).
15 / 37
ACVF and ACF for Causal ARMA(1,1)
For a causal ARMA(p,q), h ≥ 0, :

p
X q
X
2
γX (h) = φi γX (h − i) + σW θj ψj−h
i=1 j=h
Let’s consider an ARMA(1,1). When h ≥ 2, we have
γ(h) = φ1 γX (h − 1)
This means γ(h) = φh1 c for some unknown c ∈ R. To find c we

consider the “initial conditions” or when h = 0, 1.
16 / 37
ACVF and ACF for Causal ARMA(1,1)

For a causal ARMA(p,q), h ≥ 0, :
p
X q
X
2
i=1 j=h
Still considering an ARMA(1,1), when h = 0, 1, we have

1
X
2
γ(0) = φ1 γ(−1) + σW θj ψj (4)
j=0
2
γ(1) = φ1 γ(0) + σW θ1 (5)
This yields (dropping the subscripts from θ1 and φ1 )
(1 + θφ)(φ + θ) h−1
ρ(h) = φ
(1 + 2θφ + θ2 )
Notice that there is only one part dependent on h!
17 / 37
ACVF for Causal AR(1)

For a causal ACVF,
p
X q
X
2
i=1 j=h
Consider a causal AR(P) model xt = φ1 xt−1 + wt . Then

p
X
γX (h) = φi γX (h − i)
i=1
γX (h) will never cut-off to 0 for any h. This makes it difficult to
identify what p is.
18 / 37
A new idea for causal AR(p) models
We’ll need this trick in the following slide...

P∞
Note we can write Xt = i=0 ψi Wt−i
Causality means xt−2 , for example, only depends on wt−2 , wt−3 , . . .

and hence is uncorrelated with wt−1 and wt .
0
P∞ z }| {
For s > t, check Cov(ws , xt ) = i=0 ψj Cov(ws , wt−i ).
19 / 37
A new idea for causal AR(p) models
For a causal AR(1) model, the γX and ρX functions don’t zero out
past p, so instead consider
Cov(xt+2 − x̂t+2 , xt − x̂t ) = Cov(xt+2 − φxt+1 , xt − φxt+1 )

= Cov(wt+2 , xt − φxt+1 )
= Cov(wt+2 , xt ) − φCov(wt+2 , xt+1 )
=0
So, when considering the relationship between xt+2 and xt , we first

remove the linear dependence on xt+1 !
20 / 37
Notation
One way to remove linear connections is through linear regression.
Let x̂t+h denote the regression of xt+h on

{xt+h−1 , xt+h−2 , . . . , xt+1 }, which we write as
x̂t+h = β1 xt+h−1 + β2 xt+h−2 + · · · + βh−1 xt+1 . (6)
Here we do not include the intercept assuming the mean of xt is

zero. Otherwise, replace xt with xt − µx .
21 / 37
Notation
In addition, let x̂t denote the regression of xt on

{xt+1 , xt+2 , . . . , xt+h−1 }, then
x̂t = β1 xt+1 + β2 xt+2 + · · · + βh−1 xt+h−1 . (7)
22 / 37
4 Worked Examples
23 / 37
Partial Autocorrelation Function
The partial autocorrelation function (PACF) of a stationary

process xt , denoted by φhh , for h = 1, 2, . . ., is
φ11 = corr (xt+1 , xt ) = ρ(1) (8)

and
φhh = corr (xt+h − x̂t+h , xt − x̂t ), h ≥ 2. (9)
Note that, the PACF, φhh is the correlation between xt+h and xt
with the linear dependence of {xt+1 , · · · , xt+h−1 }, on each,
removed.
24 / 37
Partial Autocorrelation Function: AR(1) example
Let’s go back to the example from a few slides ago, and try to
calculate the PACF of a causal AR(1) model: xt = φxt−1 + wt ,
with |φ| < 1.
By definition, φ11 = corr (x1 , x0 ) = ρ(1) = φ.
We wrote
φ22 := Cov(xt+2 − x̂t+2 , xt − x̂t )

=0
...but why are x̂t+2 = x̂t = φxt+1 ?
25 / 37
Partial Autocorrelation Function: AR(1) example
To calculate φ22 , consider the regression of xt+2 on xt+1 , say

x̂t+2 = βxt+1 .
We choose β to minimize
h i
E (xt+2 − βxt+1 )2 = γX (0) − 2βγX (1) + β 2 γX (0).
Taking the derivative and setting that equal to 0 yields β = φ.
26 / 37
Next, consider the regression of xt on xt+1 , say x̂t = βxt+1 . We

choose β to minimize
27 / 37
In general, for a causal AR(p) model xh = pj=1 φj xh−j + wh .

P
When h > p, the regression of xh on xh−1 , . . . , x1 is
p
X
x̂h = φj xh−j .
j=1
28 / 37
Thus, when h > p, by causality,
φhh = corr (xh − x̂h , x0 − x̂0 ) = corr (wh , x0 − x̂0 ) = 0.
29 / 37
Summary
• The ACF of MA(q) model cuts off after lag q. The PACF of
an AR(p) model cuts off after lag p.
• Identification of an MA(q) model is best done with ACF;
identification of an AR(p) model is best done with PACF.
• The PACF between xt and xt−h is the correlation between
xt − x̂t and xt−h − x̂t−h . Think of it as taking the correlation
between the residuals from two regression models. The
dependence on all intermediate variables is removed.
30 / 37
4 Worked Examples
31 / 37
ACF and PACF of Causal AR(2)

AR(2)
1.0
0.5
ACF
0.0
−0.5
5 10 15 20
LAG
1.0
0.5
PACF
0.0
−0.5
5 10 15 20 32 / 37
ACF and PACF of Invertible MA(2)

MA(2)
0.2
0.0
−0.6 −0.4 −0.2
ACF
5 10 15 20
LAG
0.2
0.0
PACF
−0.6 −0.4 −0.2
5 10 15 20 33 / 37
ACF and PACF of Causal and Invertible ARMA(2,2)

ARMA(2,2)
1.0
0.5
ACF
0.0
−0.5
5 10 15 20
LAG
1.0
0.5
PACF
0.0
−0.5
5 10 15 20 34 / 37
ACF and PACF of Causal AR and Invertible MA
(From page 99, Table 3.1 of text)
AR(p) MA(q) ARMA(p,q)
ACF Decay 0 after lag q Decay

PACF 0 after lag p Decay Decay
35 / 37
Fish Population Example
This time series from “recruit.dat” contains data on fish

population in the central Pacific Ocean. The numbers represent
the number of new fish in the years 1950-1987. Question: Based
on the ACF and PACF plots, what process do you think is most
likely to describe this time series?
36 / 37
Fish Population Example

Recruitment Series
1.0
0.5
ACF
0.0
−0.5
5 10 15 20
LAG
1.0
0.5
PACF
0.0
−0.5
5 10 15 20 37 / 37

Unit 11

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Unit 11

Uploaded by

Copyright:

Available Formats

ACF for MA(q) Processes ACF for Causal ARMA(p,q) Processes Partial Autocorrelation Function Worked Examples

Unit 11: ARMA Autocorrelation and Partial

Department of Statistics, University of Virginia

Readings for Unit 11

Textbook chapter 3.3.

1 ACF for MA(q)

In this unit we will study the autocorrelation and partial

Let’s start with an MA(q) process

where we have written θ0 = 1. Then

Autocovariance for MA(q)

The autocovariance function is

Autocovariance for MA(q)

Recall that E (ws wt ) = σw2 if s = t and E (ws wt ) = 0 otherwise. So

ACF for MA(q)

1 ACF for MA(q) Processes

2 ACF for Causal ARMA(p,q) Processes

3 Partial Autocorrelation Function

ACF for Causal ARMA(p,q)

Now we discuss causal ARMA(p, q) model

We have the MA(∞) representation

Autocovariance for Causal ARMA(p,q)

It follows that E (xt ) = 0 and by (1), the autocovariance function

Autocovariance for Causal ARMA(p,q)

ACVF and ACF for Causal ARMA(p,q)

0 ≤ j < ∞, k ≥ 0, and k = j − h. Dividing through by γ(0) will

ACVF and ACF for Causal ARMA(1,1)

For a causal ARMA(p,q), h ≥ 0, :

Let’s consider an ARMA(1,1). When h ≥ 2, we have

This means γ(h) = φh1 c for some unknown c ∈ R. To find c we

ACVF and ACF for Causal ARMA(1,1)

Still considering an ARMA(1,1), when h = 0, 1, we have

ACVF for Causal AR(1)

Consider a causal AR(P) model xt = φ1 xt−1 + wt . Then

A new idea for causal AR(p) models

We’ll need this trick in the following slide...

Causality means xt−2 , for example, only depends on wt−2 , wt−3 , . . .

A new idea for causal AR(p) models

Cov(xt+2 − x̂t+2 , xt − x̂t ) = Cov(xt+2 − φxt+1 , xt − φxt+1 )

So, when considering the relationship between xt+2 and xt , we first

One way to remove linear connections is through linear regression.

Let x̂t+h denote the regression of xt+h on

x̂t+h = β1 xt+h−1 + β2 xt+h−2 + · · · + βh−1 xt+1 . (6)

Here we do not include the intercept assuming the mean of xt is

In addition, let x̂t denote the regression of xt on

x̂t = β1 xt+1 + β2 xt+2 + · · · + βh−1 xt+h−1 . (7)

1 ACF for MA(q) Processes

2 ACF for Causal ARMA(p,q) Processes

3 Partial Autocorrelation Function

Partial Autocorrelation Function

The partial autocorrelation function (PACF) of a stationary

φ11 = corr (xt+1 , xt ) = ρ(1) (8)

Partial Autocorrelation Function: AR(1) example

By definition, φ11 = corr (x1 , x0 ) = ρ(1) = φ.

φ22 := Cov(xt+2 − x̂t+2 , xt − x̂t )

...but why are x̂t+2 = x̂t = φxt+1 ?

Partial Autocorrelation Function: AR(1) example

To calculate φ22 , consider the regression of xt+2 on xt+1 , say

Taking the derivative and setting that equal to 0 yields β = φ.

Partial Autocorrelation Function

Next, consider the regression of xt on xt+1 , say x̂t = βxt+1 . We

Partial Autocorrelation Function

In general, for a causal AR(p) model xh = pj=1 φj xh−j + wh .

Partial Autocorrelation Function