Statistics 203: Introduction to Regression

and Analysis of Variance

Time Series: Brief Introduction

Jonathan Taylor

- p. 1/15

Todays class

Todays class
Modelling correlation
Other models of correlation
Autoregressive models
AR(1),
AR(1),

= 0.95
= 0.5

AR(k) models

=
0.9, 2 = 0.2

AR(2), 1

Moving average &

Models for time-correlated noise.

Stationary time series.
ARMA models.
Autocovariance, power spectrum.
Diagnostics.

ARM A(p, q) models

ARM A(2, 4)
Stationary time series
Estimating autocovariance /
correlation
Estimating power spectrum
Diagnostics

- p. 2/15

Modelling correlation

Todays class

Modelling correlation

In the mixed effects model

Other models of correlation

Y = X + Z +

Autoregressive models

= 0.95
AR(1), = 0.5
AR(k) models
AR(2), 1 =
0.9, 2 = 0.2
AR(1),

with N (0, 2 I) and N (0, D) we were essentially

saying
Y N (X, ZDZ t + 2 I)

Moving average &

ARM A(p, q) models

ARM A(2, 4)
Stationary time series

Estimating autocovariance /
correlation
Estimating power spectrum
Diagnostics

We then estimated D from the data (more precisely, R does

this for us).
We can impose structure on D if necessary. For example, in
two-way random effects ANOVA, we assumed that
i , j , ()ij were independent mean zero normal random
variables.
In summary, a mixed effect model can be thought of as
modelling the correlation in the errors of Y coming from
sampling from a population.

- p. 3/15

Other models of correlation

Todays class
Modelling correlation
Other models of correlation
Autoregressive models
AR(1),
AR(1),

= 0.95
= 0.5

AR(k) models

=
0.9, 2 = 0.2

AR(2), 1

Moving average &

ARM A(p, q) models

ARM A(2, 4)
Stationary time series
Estimating autocovariance /
correlation
Estimating power spectrum
Diagnostics

Not all correlations come from sampling.

Another common source is correlation in time.
Example: imagine modelling monthly temperature in a given
location over many years.
Yt = t%12 + t , 1 t T
Clearly, will vary smoothly as a function of t, but there
will also be correlation in t due to weather systems that
last more than one day.
To estimate optimally and (especially to) make
inferences about we should take these correlations into
account.
Time series models are models of such (auto)correlation.
Good references: Priestley, Spectral Theory and Time
Series; Brockwell and Davis, Introduction to Time Series
and Forecasting.
Nottingham temperature example.
Today we will just talk about time series in general.
- p. 4/15

Autoregressive models

Todays class

Modelling correlation

Simplest stationary autocorrelation

Other models of correlation

t = t1 + t

Autoregressive models

= 0.95
AR(1), = 0.5
AR(k) models
AR(2), 1 =
0.9, 2 = 0.2
AR(1),

Moving average &

ARM A(p, q) models

ARM A(2, 4)

Stationary time series

Estimating autocovariance /
correlation
Estimating power spectrum

Diagnostics

where N (0, 2 I) are i.i.d. Normal random variables,

|| < 1.
This is called an auto-regressive process: auto because
is like a regression of on its past.
It is called AR(1) because it only goes 1 time point into the
past.
Covariance / correlation function
2 |j|
,
Cov(t , t+j ) =
2
1

Cor(t , t+j ) = |j|

Model is stationary because Cov(t , t+j ) depends only on

|j|.

- p. 5/15

2
0
2
4
6

ar1.sim1

AR(1), = 0.95

50

100
Time

150

200
- p. 6/15

0
1
2

ar1.sim2

AR(1), = 0.5

50

100
Time

150

200
- p. 7/15

AR(k) models

Todays class
Modelling correlation
Other models of correlation
Autoregressive models
AR(1),
AR(1),

= 0.95
= 0.5

AR(k) models

j=1

=
0.9, 2 = 0.2

AR(2), 1

Moving average &

ARM A(p, q) models

ARM A(2, 4)
Stationary time series

The AR(1) model can be easily generalized to the AR(p)

model:
p
X
j tj + t
t =

where N (0, 2 I) are i.i.d. Normal random variables.

Condition on s: all roots of the (complex) polynomial

Estimating autocovariance /
correlation
Estimating power spectrum
Diagnostics

(z) = 1

p
X

j z j

j=1

are within the unit disc in the complex plane.

- p. 8/15

0
2
4

ar2.sim

AR(2), 1 = 0.9, 2 = 0.2

50

100
Time

150

200
- p. 9/15

Moving average & ARM A(p, q) models

Todays class
Modelling correlation

M A(q) is another stationary model:

Other models of correlation

Autoregressive models
AR(1),
AR(1),

= 0.95
= 0.5

t =

AR(k) models

Moving average &

Stationary time series

Estimating autocovariance /
correlation
Estimating power spectrum

j tq

j=0

=
0.9, 2 = 0.2

AR(2), 1

ARM A(p, q) models

ARM A(2, 4)

q
X

where N (0, 2 I) are i.i.d. Normal random variables.

No conditions on s this is always stationary.
ARM A(p, q) model:

Diagnostics

t =

p
X
l=1

l Xtl +

q
X

j tq

j=0

where N (0, 2 I) are i.i.d. Normal random variables.

- p. 10/15

0
20

arma.sim

20

40

ARM A(2, 4)

50

100
Time

150

200
- p. 11/15

Stationary time series

Todays class
Modelling correlation
Other models of correlation
Autoregressive models
AR(1),
AR(1),

= 0.95
= 0.5

AR(k) models

=
0.9, 2 = 0.2

AR(2), 1

Moving average &

ARM A(p, q) models

ARM A(2, 4)
Stationary time series
Estimating autocovariance /

correlation
Estimating power spectrum
Diagnostics

In general a (Normally distributed) time series (t ) is

stationary if
Cov(t , t+j ) = R(|j|)
for some covariance function R.
If errors are not normally distributed then the process is
called weakly stationary, or stationary in mean-square.
The function R(t) can generally be expressed as the Fourier
transform of a spectral density
Z
1
R(t) =
eit fR () d
2
where f is called the spectral density of the process.
Conversely
X
eit R(t)
fR (t) =
t

The function fR is sometimes called the power spectrum of

.

- p. 12/15

Estimating autocovariance / correlation

Todays class
Modelling correlation
Other models of correlation
Autoregressive models
AR(1),
AR(1),

= 0.95
= 0.5

nt

X
1
b =
(j+t )(j ).
R(t)
n j=1

AR(k) models

=
0.9, 2 = 0.2

AR(2), 1

Moving average &

ARM A(p, q) models

ARM A(2, 4)
Stationary time series
Estimating autocovariance /
correlation
Estimating power spectrum
Diagnostics

Natural estimate of covariance function for t 0 based on

observing (1 , . . . , n )

Estimate of correlation function

b
R(t)
d =
.
Cor(t)
b
R(0)

- p. 13/15

Estimating power spectrum

Todays class

Estimate of power spectrum based on observing (1 , . . . , n )

is called the periodogram, the discrete Fourier transform of

P n
it 2

t
t=1 e
I() =
n

In fact

Modelling correlation
Other models of correlation
Autoregressive models
AR(1),
AR(1),

= 0.95
= 0.5

AR(k) models

=
0.9, 2 = 0.2

AR(2), 1

Moving average &

ARM A(p, q) models

ARM A(2, 4)

I() =

Stationary time series

Estimating autocovariance /

X
t

correlation
Estimating power spectrum
Diagnostics

it
b
R(t)e

b
i.e. it is the Fourier transform of R(t).
It is customary to use a smoothed periodogram as an
estimate of fR
Z
fbR () = Kh (( )/h)I() d.

for some kernel Kh .

If s are i.i.d. (hence stationary), then

E(I()) = Var().

- p. 14/15

Diagnostics

Todays class
Modelling correlation
Other models of correlation
Autoregressive models
AR(1),
AR(1),

= 0.95
= 0.5

AR(k) models

Suppose we fit an ARM A(p, q) model to observations

(1 , . . . , n ): how can we tell if the fit is good?
How do we do this? By residuals of course. In an AR(p)
model, for instance, define

=
0.9, 2 = 0.2

AR(2), 1

Moving average &

bt = Xt

ARM A(p, q) models

ARM A(2, 4)
Stationary time series
Estimating autocovariance /
correlation
Estimating power spectrum

Diagnostics

p
X
j=1

bj Xtj .

These should look like an i.i.d. sequence, at least roughly.

Can plot residuals themselves, autocorrelation function of
residuals, and cumulative periodogram.

- p. 15/15