Intro Stat 153

Logistics
Introduction to Time Series

Statistical Models of Time Series
Stat 153: Introduction to Time Series
Joan Bruna
Department of Statistics
UC, Berkeley
September 3, 2015
Joan Bruna Stat 153: Introduction to Time Series

Logistics
Instructor: Joan Bruna Estrach, joan.bruna@berkeley.edu

Office hours and location: Tuesdays 2pm-4pm, Evans 419.
bCourses link:
https://bcourses.berkeley.edu/courses/1364346
GSIs: To be announced...

Logistics
Grading
Midterm Exams: October 13th, November 12th, 15 % each.

Final Exam: December 18th, 35 %.
Lab/Homework assignments1 . (every one or two weeks), 25
%
Lab Final Project, 10 %.
1
lowest grade will be dropped
Logistics
What is a Time Series?

Logistics

Logistics

S&P 500 index
Statistical Model necessary.

Logistics

S&P 500 index, zoomed
Interaction of random and non-random behavior.

Logistics

Logistics
trend
seasonality (periodicity)
Logistics

Logistics
Periodic, stationary phenomena is studied with spectrograms.

!N !Ti on the same location (see Figure 1), initially used for the
i i
t=1 P (qt = j|O , λm )Ot training of traffic conflict observers in the 1980s. Their
µcj = !
i=1
N !Ti
(1)
i=1
i
t=1 P (qt = j|O , λm )
length ranges from 10 seconds to 60 seconds. Despite the
videotape aging, the approximate alignment of the field of
!N !Ti i i c i c T
Logistics
view between sequences and occasional camera jitter, it
P (qt = j|O , λm )(Ot − µj )(Ot − µj )
Σcj = i=1 t=1!N !Ti
i, λ )
Introduction to used
could be digitized and Time to test Series
our method. The training
i=1 t=1 P (qt = j|O m program highlights nine traffic conflicts in nine sequences
Statistical
(2) Models of Time Series
(see Figure 2). However, only five of them are used as
The parameters of the HMM λm can then be adapted using
training instances. Another one (fourth sequence) is not used
the following equations:
because the vehicles are not detected anymore at the time
′ they are really in conflict, and the rest of the traffic conflicts
µj = (1 − α)µj + αµcj (3)
involve pedestrians and cyclists that are very difficult to

′
Σj =
′
′
′
(1 − α)(Σj + (µj − µj )(µj − µj )T )
′
distinguish and track on account of the videotape quality.
There is no traffic conflict highlighted in the last sequence. It
is sometimes difficult to judge the performance of the method
c c c T
+α(Σj + (µj − µj )(µj − µj ) ) (4) since there is no ground truth, apart from the traffic conflicts
′ ′ used in the program.
where µj and Σj are the mean and covariance of the
Gaussian of the state Sj of the adapted HMM. α is a weight-
ing factor that controls the balance between the original
model and the new estimates on the trajectories involved
in the training traffic conflicts. Both the original model and
the adapted model are kept in the set of HMMs used for
detection.
A traffic conflict is an interaction, defined as an obser-
vational situation in which two or more vehicles are close
enough in space and time, and are nearing each other. So
far, the method has built a set of HMMs, among which
some are models of conflicting trajectories. But a trajectory
is conflicting only with respect to another one. It is their
conjunction in time and space that creates a danger of
Fig. 1. An image of the traffic sequences.
collision. Therefore, the models of the conflicting trajectories
(to which the trajectories involved in the training traffic
conflict instances are assigned) are memorized by pairs (e.g.
pairs of models 4 and 7, 11 and 2, 3 and 7). The traffic
conflict detection proceeds as follows:
1) Vehicles are tracked.
2) If two vehicles are close enough (threshold on their
distance) and nearing each other (their distance de-
creases), an interaction is detected.
3) Each interacting vehicle trajectory is assigned to a
HMM.
4) If the HMMs of both interacting trajectories were both
memorized as conflicting (e.g. any of the pairs of
models 4 and 7, 11 and 2, 3 and 7), a traffic conflict
between these two vehicles is detected.
Fig. 2. An example of trajectories involved in a traffic conflict.
V. E XPERIMENTAL E VALUATION from Saunier and Sayed
A simple vehicle detection and tracking algorithm is used, All K HMMs of a mixture have the same structure param-
based on the implementation of the KLT feature tracking eter values (number of states, simple Gaussian observation
algorithm of [8], used in [9]. The advantages of feature- 2 http://www.cs.ubc.ca/∼murphyk/Software/HMM/hmm.
Robotics
based algorithms include the abilities to work well under html
Control Theory (Kalman)

Self-driving cars ...
Logistics

Logistics
State space models (language).

Logistics
State space models (language).

Prediction, compression, ...
Logistics
This is not an IID world
Fundamental characteristic of time-series: in general, samples

are correlated (thus statistically dependent).

Logistics
This is not an IID world
Fundamental characteristic of time-series: in general, samples

are correlated (thus statistically dependent).
Estimating, modeling and analyzing this dependency/correlation is

the main objective of this course.

Logistics
Objectives of Time Series Analysis
Compact description of the data: statistical modeling.

Interpretation
Forecasting/Prediction
Control
Simulation
Hypothesis testing

Logistics
Example: Monthly Sales of a Souvenir Shop

(from P.Bartlett’s slides, Makridakis, Wheelwright and Hyndman, 1998)

Logistics
Transformed
Example: Transformation of the data
data
12
11.5
11
10.5
10
9.5
8.5
7.5
7
0 10 20 30 40 50 60 70 80 90
Logistics
Example: Trend Trend
12
11.5
11
10.5
10
9.5
8.5
7.5
7
0 10 20 30 40 50 60 70 80 90
Logistics
Residuals
Example: Look at the residuals
1.5
0.5
−0.5
−1
0 10 20 30 40 50 60 70 80 90
Logistics
Trend Seasonality
Example: Modeling and seasonaland
variation
Trend
12
11.5
11
10.5
10
9.5
8.5
7.5
7
0 10 20 30 40 50 60 70 80 90
Logistics
Example
Compact Description: Decomposition model
Xt = Tt + St + Rt .

Logistics
Example
Xt = Tt + St + Rt .
This model is interpretable. (eg seasonal adjustment due to

holidays).

Logistics
Example
Xt = Tt + St + Rt .
This model is interpretable. (eg seasonal adjustment due to

holidays).
Forecasting/Prediction: Expected Sales next month?

Logistics
Overview of the Course
1 Time series basics.

2 Time domain Models.
(Midterm 1)
3 Spectral and Wavelet Analysis.
(Midterm 2)
4 State Space and Discrete Models.

Logistics
Time Series Basics
Stationarity.
Autocorrelation Function.
Chasing Stationarity.

Logistics
Time domain Models
Auto-Regressive models (AR, MA, ARMA, ...)

Local Volatility models (GARCH, ..)
ARIMA models
Forecasting
Parameter Estimation

Logistics
Spectral Methods
Correlation is a form of (global) smoothness. How to study
smoothness properties?

Logistics
Spectral Methods
Joseph Fourier (1768-1830).

Logistics
Spectral Methods
Joseph Fourier (1768-1830). Smoothness is studied

mathematically using Spectral (or Fourier, or Harmonic) Analysis.
Several Spectral Representations possible (Fourier, Windowed
Fourier, Wavelet, etc.)
Logistics
Spectral and Wavelet Analysis
Spectral Density.
Time-Frequency Representations, Periodogram.
Introduction to Wavelet Analysis
Spectral Estimation.
Applications to speech recognition.

Logistics
State Space and Discrete Models
ARMAX models
Hidden Markov models (HMM).
Kalman Filter.
Recurrent Neural Networks (RNN). (time allowing)

Important Examples
Statistical Measurements
Logistics
Stationarity
Estimation of Correlation
Sample ACF
Prediction with ACF
Time Series Modeling/Notation
How do we define a Time Series?

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

A (stochastic) Time Series is a collection {Xt } of random variables
indexed by a temporal index t.
In this course, we will mostly consider discrete time series:

t = 1, 2, 3, . . . is a discrete index.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

In this course, we will mostly consider discrete time series:

t = 1, 2, 3, . . . is a discrete index.
{Xt } will always denote a stochastic process.
{xt } will always denote a particular realization.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Time Series Modeling
How do we specify a Time Series Model?

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Time Series Modeling
How do we specify a Time Series Model?
A Time Series model is (fully) specified by giving the joint

distribution of {Xt }:
P(X1 ≤ x1 , X2 ≤ x2 , . . . , Xt ≤ xt ) for all t, x1 , . . . , xt .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
The Curse of Dimensionality
As t increases, the complexity of the previous model grows

exponentially.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

exponentially.
Intractable in general.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

exponentially.
We will resort to low-order statistics only (mostly first and
second order).

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
White Noise
{Xt } is a white noise if for all t,
1 E (Xt ) = 0,
2 var (Xt ) = σ 2 ,
3 Xt and Xu are uncorrelated for t 6= u.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
White Noise
1 E (Xt ) = 0,
2 var (Xt ) = σ 2 ,
In particular, if {Xt } are i.i.d with zero mean, {Xt } is a white
noise. Also,
Y
P(X1 ≤ x1 , . . . , Xt ≤ xt ) = P(Xi ≤ xi ) .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
White Noise
1 E (Xt ) = 0,
2 var (Xt ) = σ 2 ,
noise. Also,
Y
P(X1 ≤ x1 , . . . , Xt ≤ xt ) = P(Xi ≤ xi ) .
Forecasting is not possible under iid noise:
P(Xt ≤ xt | X1 , . . . , Xt−1 ) = P(Xt ≤ xt ) .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Most Famous Time Series
Gaussian White Noise: Xt ∼ N (0, σ 2 ) iid.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Most Famous Time Series
Gaussian White Noise: Xt ∼ N (0, σ 2 ) iid.
It cannot model any time-dependencies.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Bernouilli White Noise
Xt ∼ Bern(p), with p ∈ [0, 1].

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Bernouilli White Noise
Xt ∼ Bern(p), with p ∈ [0, 1].
Eg: models the success at a casino roulette.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Moving Averages
A simple way to model dependencies across samples is to average

across time.
If {Wt } is white noise, the series
∆
X
Xt = λk Wt+k
k=−∆
is called a moving average (MA(2∆)).

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Moving Averages
P∆
Xt = k=−∆ λk Wt+k .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Random Walks
P
Consider {Wt } white noise, and Xt = i≤t Wi .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Random Walks
P
Differences ∇Xt = Xt − Xt−1 = Wt .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Random Walks
E (Xt ) = , var (Xt ) =

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Random Walks
E (Xt ) = 0 , var (Xt ) = tσ 2 .
Variance increases with time!

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Random Walks
Recall S&P data.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Random Walks
Recall S&P data.
Random walks and their generalizations (Brownian Motions) are

good financial models.
Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Signal with Noise

In many problems, good models combine deterministic with
stochastic aspects:
Xt = F (t) + Wt , with F deterministic .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Signal with Noise

In many problems, good models combine deterministic with
stochastic aspects:
Xt = F (t) + Wt , with F deterministic .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Last Lecture Review


Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Last Lecture Review



Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Last Lecture Review


We will resort to low-order statistics only (mostly first and second
order).

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Review: White Noise

1 E (Xt ) = 0,
2 var (Xt ) = σ 2 ,

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Review: White Noise

1 E (Xt ) = 0,
2 var (Xt ) = σ 2 ,
noise. Also,
Y
P(X1 ≤ x1 , . . . , Xt ≤ xt ) = P(Xi ≤ xi ) .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Review: White Noise

1 E (Xt ) = 0,
2 var (Xt ) = σ 2 ,
noise. Also,
Y
P(X1 ≤ x1 , . . . , Xt ≤ xt ) = P(Xi ≤ xi ) .
Forecasting is not possible under iid noise:
P(Xt ≤ xt | X1 , . . . , Xt−1 ) = P(Xt ≤ xt ) .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Review: Random Walks
P

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Review: Random Walks
P
Differences ∇Xt = Xt − Xt−1 = Wt .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Review: Mean and Covariance of a Time Series
Definition
The mean function of a time series {Xt } is
µX (t) = E (Xt ) .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Review: Mean and Covariance of a Time Series
Definition
The mean function of a time series {Xt } is
µX (t) = E (Xt ) .
Definition
The autocovariance function of a time series {Xt } is
RX (t, s) = cov(Xt , Xs ) = E ((Xt − E (Xt ))(Xs − E (Xs ))) .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Lecture 2
Autocovariance examples
Autocorrelation and Cross-Correlation
Stationarity
Linear Processes

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Examples
P
{Xt } Random Walk: Xt = i≤t Wi , with {Wt } iid white noise.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Examples
P
 
X X
µX (t) = E (Xt ) = E  Wi  = E (Wi ) = 0 .
i≤t i≤t

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Examples
P
 
X X
µX (t) = E (Xt ) = E  Wi  = E (Wi ) = 0 .
i≤t i≤t
X X X
RX (s, t) = cov( Wi , Wi 0 ) = cov(Wi , Wi 0 ) = min(s, t)σ 2 .
i≤s i 0 ≤t i≤s,i 0 ≤t

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Examples
{Xt } = F (t) + {Wt }: signal with white noise.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Examples
µX (t) = E (Xt ) = E (F (t) + Wt ) = F (t) .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Examples
µX (t) = E (Xt ) = E (F (t) + Wt ) = F (t) .

σ2 if s=t ,
RX (s, t) = cov(F (s)+Ws , F (t)+Wt ) = cov(Ws , Wt ) =
0 otherwise .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Last Important Example
P∆
Moving average process: Xt = k=−∆ Wt+k , with {Wt } white
noise.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
P
Moving average process: Xt = ∆ k=−∆ Wt+k , with {Wt } white
noise.
RX (s, t) =? .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
If |t − s| > 2∆:
∆ ∆
!
X X
RX (s, t) = cov Ws+k , Wt+k =
k=−∆ k=−∆

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
If |t − s| > 2∆:
∆ ∆
!
X X
RX (s, t) = cov Ws+k , Wt+k =0.
k=−∆ k=−∆

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
If |t − s| ≤ 2∆:
∆ ∆
!
X X
RX (s, t) = cov Ws+k , Wt+k =
k=−∆ k=−∆

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
If |t − s| ≤ 2∆:
∆ ∆
!
X X
RX (s, t) = cov Ws+k , Wt+k = (2∆ − |t − s|)σ 2 .
k=−∆ k=−∆

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
If |t − s| ≤ 2∆:
∆ ∆
!
X X
RX (s, t) = cov Ws+k , Wt+k = (2∆ − |t − s|)σ 2 .
k=−∆ k=−∆
In particular, RX only depends upon the difference t − s.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Autocorrelation function (ACF)
In some situations, it is better to normalize the autocovariance.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

Definition
The Autocorrelation function (ACF) of a time series {Xt } is
defined as
RX (s, t)
ρX (s, t) = p .
RX (s, s)RX (t, t)

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

Definition
The Autocorrelation function (ACF) of a time series {Xt } is
defined as
RX (s, t)
ρX (s, t) = p .
RX (s, s)RX (t, t)
Measures linear predicability of Xt from Xs .

It satisfies
−1 ≤ ρX (s, t) ≤ 1 .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Cross-Covariance and Cross-Correlation

We might also want to measure predictability from one series {Xt }
to another {X̃t }.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

to another {X̃t }.
Exxon vs Shell:

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

to another {X̃t }.
Exxon vs Google:

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

to another {X̃t }.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

to another {X̃t }.
Definition
The Cross-Covariance between {Xt } and {X̃t } is
RX ,X̃ (s, t) = cov(Xs , X̃t ) ,
and the cross-correlation is

RX ,X̃ (s, t)
ρX ,X̃ (s, t) = .
RX (s, s)RX̃ (t, t)

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Stationarity: Motivation
Q: Main difference between figures?

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Strict Stationarity
Definition
A Time Series {Xt } is strictly Stationary if
P(Xt1 ≤ c1 , Xt2 ≤ c2 , . . . , Xtk ≤ ck ) =

P(Xt1 +h ≤ c1 , Xt2 +h ≤ c2 , . . . , Xtk +h ≤ ck ) , ∀ k, ti , ci , h .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Strict Stationarity
Definition
P(Xt1 ≤ c1 , Xt2 ≤ c2 , . . . , Xtk ≤ ck ) =

Time is relative: statistical properties do not depend upon

time reference.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Strict Stationarity
Definition
P(Xt1 ≤ c1 , Xt2 ≤ c2 , . . . , Xtk ≤ ck ) =


time reference.
In particular, using k = 1, P(Xt ≤ c) is independent of t.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Strict Stationarity
Definition
P(Xt1 ≤ c1 , Xt2 ≤ c2 , . . . , Xtk ≤ ck ) =


time reference.
In particular, using k = 1, P(Xt ≤ c) is independent of t.
Far LESS parameters to describe the process.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Strict Stationarity: Consequences

Using previous property for k = 1, P(Xt ≤ c) = P(Xs ≤ c)
for all t, s implies
µX (t) = cte .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

µX (t) = cte .
Using previous property for k = 2, joint law (Xt , Xs ) = joint

law (Xt+h , Xs+h ), hence
∀ h , RX (t, s) = RX (t + h, s + h) =⇒ RX depends only on |t − s| .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

µX (t) = cte .
Using previous property for k = 2, joint law (Xt , Xs ) = joint

law (Xt+h , Xs+h ), hence
∀ h , RX (t, s) = RX (t + h, s + h) =⇒ RX depends only on |t − s| .
(since R(0, t − s) = R(s, t) = R(t, s) = R(0, s − t)) .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Weak Stationarity
In practice, strict Stationarity is hard to impose/estimate.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Weak Stationarity

Definition
A Time series {Xt } is weakly stationary if it has finite variance,
µX (t) = cte and RX (t, s) = f (|t − s|).

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Weak Stationarity

Definition
If {Xt } is strictly stationary (and has finite variance), then it

is weakly stationary.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Weak Stationarity

Definition

Converse not true, but ...

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Weak Stationarity

Definition

Converse not true, but ...
...If {Xt } is Gaussian and weakly stationary, then it is strictly
stationary.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Autocorrelation Function (ACF)
The Autocorrelation of a stationary process {Xt } is
RX (h) cov(Xt , Xt+h )

ρX (h) = = .
RX (0) var (Xt )

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Examples: White Noise
{Wt } white noise.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Examples: White Noise
{Wt } white noise. It is weakly stationary since
µw (t) = 0 ∀ t , and Rw (s, t) = σ 2 1(|s − t| = 0) .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Examples: Random Walk
{Xt } Random Walk. We saw that
µX (t) = 0 , RX (s, t) = σ 2 min(s, t) .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Examples: Random Walk
{Xt } Random Walk. We saw that
µX (t) = 0 , RX (s, t) = σ 2 min(s, t) .
Thus it is NOT stationary.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Examples: MA(1) process
Moving average process MA(1):
Xt = Wt + λWt−1 , {Wt } white noise .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
We have µX (t) = 0, and
RX (t, t + h) = E (Xt Xt+h )

= E ((Wt + λWt−1 )(Wt+h + λWt+h−1 ))
 2
 σ (1 + λ2 ) if h = 0
= σ 2 λ if h = ±1

0 otherwise .
Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
We have µX (t) = 0, and
RX (t, t + h) = E (Xt Xt+h )

= E ((Wt + λWt−1 )(Wt+h + λWt+h−1 ))
 2
 σ (1 + λ2 ) if h = 0
= σ 2 λ if h = ±1

0 otherwise .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Examples: MA(1) Process.
ACF of {Xt }:

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Last Important Example: Autoregressive Process

Autoregressive AR(1) process:
Xt = λXt−1 + Wt , with {Wt } white noise and |λ| < 1 .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

By substituting the recursion we obtain
Xt = Wt + λWt−1 + λ2 Wt−2 + λ3 Wt−3 + . . .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

Xt = Wt + λWt−1 + λ2 Wt−2 + λ3 Wt−3 + . . .
∞
!
X X
µX (t) = E λk Wt−k = λk E (Wt−k ) = 0 , and
k=0 k
∞
X σ2
E Xt2 = λ2k σ 2 = .
1 − λ2
k=0

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Autoregressive Process
Suppose h > 0 first. Then
RX (t, t + h) = cov(Xt , Xt+h ) = cov(Xt , λXt+h−1 + Wt+h )

= λcov(Xt , Xt+h−1 )
= λh cov(Xt , Xt )
λ|h| σ 2
=
1 − λ2
(check for h < 0 at home).

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

= λh cov(Xt , Xt )
λ|h| σ 2
=
1 − λ2
So AR(1) is (weakly) stationary.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Example: AR(1) Process.
ACF of {Xt }:

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Review: Stationarity and Correlation
A time series {Xt } has mean function µt = E (Xt ) and

autocovariance
RX (t, t + h) = E ((Xt − µt )(Xt+h − µt+h )) .
It is (weakly) stationary if both functions are independent of t. We

then write µ and RX (h).
The Autocorrelation Function (ACF) of {Xt } is
RX (h)
ρX (h) = .
RX (0)

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Example: Autoregressive Process


Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

Xt = Wt + λWt−1 + λ2 Wt−2 + λ3 Wt−3 + . . .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

Xt = Wt + λWt−1 + λ2 Wt−2 + λ3 Wt−3 + . . .
∞
!
X X
µX (t) = E λk Wt−k = λk E (Wt−k ) = 0 , and
k=0 k
∞
X σ2
E Xt2 = λ2k σ 2 = .
1 − λ2
k=0

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

= λh cov(Xt , Xt )
λ|h| σ 2
=
1 − λ2

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

= λh cov(Xt , Xt )
λ|h| σ 2
=
1 − λ2
So AR(1) is (weakly) stationary.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Example: AR(1) Process.
ACF of {Xt }:

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Today’s menu
Linear Processes
Estimation of Autocovariance and ACF.
Prediction with ACF.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Linear Processes
So far, all stationary processes we have seen are of the form

∞
X
Xt = µ + ψk Wt−k ,
k=−∞
with
{Wt } white noise.
P 2
k |ψk | < ∞.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Linear Processes
So far, all stationary processes we have seen are of the form

∞
X
Xt = µ + ψk Wt−k ,
k=−∞
with
{Wt } white noise.
P 2
k |ψk | < ∞.
These are called linear processes.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Linear Processes
Proposition
Any linear Process {Xt } is weakly stationary, with
µX (t) = µ ,
∞
X
RX (h) = σ 2 ψk ψk+h .
k=−∞

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Linear Processes
Proposition
Any linear Process {Xt } is weakly stationary, with
µX (t) = µ ,
∞
X
RX (h) = σ 2 ψk ψk+h .
k=−∞
Also, stationary processes are essentially linear (outside the scope

of stat153).

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Examples of Linear Processes
For white noise, choose µ = 0 and

1 if k = 0 ,
ψk =
0 otherwise.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
For a MA(1) process, choose µ = 0 and


 1 if k = 0 ,
ψk = λ if k = 1 ,

0 otherwise.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
For a AR(1) process, choose µ = 0 and

k
λ if k ≥ 0 ,
ψk =
0 otherwise.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Q: What about a random walk?

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Q: What about a random walk?

X X
Xt = Wt−k 6= ψk Wt−k .
0≤k≤t k

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
A slide for EE
We can view a linear process {Xt } as

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
A slide for EE
We can view a linear process {Xt } as
{Xt } is thus obtained by filtering a white noise with the filter

ψ.
The filter operation is known also as convolution:
X
(W ∗ ψ)t := ψk Wt−k .
k

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Key quantities to estimate
Suppose {Xt } is a stationary process. Given a finite number of

observations x1 , . . . , xn , we need to estimate
Its mean µ.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

Its mean µ.
Its autocovariance function
RX (h) = cov(Xt , Xt+h ) ,

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

Its mean µ.
Its autocovariance function
RX (h) = cov(Xt , Xt+h ) ,
and its ACF:

RX (h)
ρX (h) = .
RX (0)

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Sample Mean and Autocorrelation
The sample mean is

n
1X
b=
µ xi .
n
i=1

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
The sample mean is

n
1X
b=
µ xi .
n
i=1
The sample autocorrelation is

n−h
1X
Rc
X (h) = (xi − µ
b)(xi+h − µ
b) , (for h < n) .
n
i=1

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
The sample mean is

n
1X
b=
µ xi .
n
i=1

n−h
1X
Rc
X (h) = (xi − µ
b)(xi+h − µ
b) , (for h < n) .
n
i=1
The sample ACF is

Rc
X (h)
ρc
X (h) = .
RcX (0)
Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Sample Mean
Q: What is the variance of the sample mean estimator?

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Sample Mean
Q: What is the variance of the sample mean estimator?

1 X X
µ) =
var (b 2
cov( xi , xi 0 )
n 0
i i
1
= 2 (nRX (0) + (n − 1)(RX (1) + RX (−1)) + . . .
n
+(n − 2)(RX (2) + RX (−2)) + . . . (RX (n − 1) + RX (1 − n)))
n
1 X |h|
= 1− RX (h) .
n n
h=−n
Consequence: If RX (h) is smooth, variance of the sample mean

increases.
Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Sample Autocovariance
n−h
1X
Rc
X (h) = (xi − µ
b)(xi+h − µ
b) , (for h < n) .
n
i=1

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
n−h
1X
Rc
X (h) = (xi − µ
b)(xi+h − µ
b) , (for h < n) .
n
i=1
Similar to the sample covariance of (x1 , x1+h ), . . . , (xn−h , xn ),

except
divide by n instead of n − h,
b using all n samples.
sample mean µ
Why?

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Define a linear combination using samples of a stationary process
{Xt }:
Y = a1 X1 + a2 X2 + . . . an Xn .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
{Xt }:
Y = a1 X1 + a2 X2 + . . . an Xn .
Then
var (Y ) = cov(Y , Y )
Xn
= ai ai 0 RX (i − i 0 )
i,i 0 =1
≥ 0 , ( ∀a) .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
{Xt }:
Y = a1 X1 + a2 X2 + . . . an Xn .
Then
var (Y ) = cov(Y , Y )
Xn
= ai ai 0 RX (i − i 0 )
i,i 0 =1
≥ 0 , ( ∀a) .
P
A function (or matrix) R such that i,i 0 ai ai 0 Ri,i 0 ≥ 0 for all a is
called positive semi-definite.
Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Proposition
The Autocovariance is positive semi-definite.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Proposition
Proposition
The Sample Autocovariance is positive semi-definite.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Proposition
Proposition
The Sample Autocovariance is positive semi-definite.
If we divide by n − h instead it is no longer true.

The same holds for the ACF ρX and the sample ACF ρˆX .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Example
Consider {Wt } white gaussian noise, and
Xt = 2 + Wt − Wt−1 .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Example
Consider {Wt } white gaussian noise, and
Xt = 2 + Wt − Wt−1 .
We can derive that


 1 if h = 0 ,
ρX (h) = − 21 if |h| = 1 ,

0 otherwise.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Example
Samples from {Xt }:

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Example
Sample ACF ρc
X (h):

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Example
Sample ACF ρc
X (h) with true ACF (in red):

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Example
Sample ACF ρc
X (h) with true ACF (in red):
Q: How to assess presence/absence of correlation?

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Large Sample Distribution of ACF
Proposition
Let {Wt } be a white noise and fix H > 0. Then, under mild
assumptions on the law of Wt , for large n, the sample ACF ρc X (h),
h = 1, . . . , H, is approximately Normally distributed, with zero
mean and variance 1/n.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Large Sample Distribution of ACF
Proposition
Let {Wt } be a white noise and fix H > 0. Then, under mild
assumptions on the law of Wt , for large n, the sample ACF ρc X (h),
h = 1, . . . , H, is approximately Normally distributed, with zero
mean and variance 1/n.
Consequence: Under white noise assumption,

√
ρX (h)| > 1.96/ n) ≈ 0.05.
P(|c

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Example
Sample ACF ρc
X (h) with true ACF (in red) and significance region
(in green):

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Review: Sample Autocovariance

The sample mean is
n
1X
b=
µ xi .
n
i=1

n−h
1X
Rc
X (h) = (xi − µ
b)(xi+h − µ
b) , (for h < n) .
n
i=1
The sample ACF is
Rc
X (h)
ρc
X (h) = .
RcX (0)

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Sample ACF
Many properties of non-white processes (even non-stationary) are

revealed through the sample ACF:

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Sample ACF
Many properties of non-white processes (even non-stationary) are

revealed through the sample ACF:
{Xt } Sample ACF

white 0
Trend Slow Decay
Periodic Periodic
MA(q) zero for |h| > q
AR(p) exponential decay

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
White noise ACF

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Trend
Let {Xt } be a stationary process. If
Yt = Xt + βt , with β 6= 0 ,
then {Yt } has trend (or also, drift).

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Trend
Yt = Xt + βt , with β 6= 0 ,

We can also replace the linear function βt with more complicated
trends (eg quadratic).

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Trend
Yt = Xt + βt , with β 6= 0 ,

We can also replace the linear function βt with more complicated
trends (eg quadratic).
Q: Can we detect trend via the estimate of the ACF?

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Trend and ACF

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Periodicity and ACF

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Sample ACF for MA(1)
Xt = 0.4Wt−1 + Wt

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Sample ACF for MA(1)
Xt = 0.4Wt−1 + Wt

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Sample ACF for AR(1)
Xt = 0.7Xt−1 + Wt

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Sample ACF for AR(1)
Xt = 0.7Xt−1 + Wt

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Linear Prediction in Time Series
Q: For a given time series model, how well can Xt be predicted

from the past?

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Linear Prediction in Time Series
Q: For a given time series model, how well can Xt be predicted

from the past?
Q2: How is the prediction related to the Autocorrelation function?

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Example: AR(1) process

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

Scatterplot between Xt and Xt−1 :

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF


Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF


Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF


Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Recall the ACF function:
ACF controls quality of linear prediction.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
Least Squares and ACF
Least squares estimate of a random variable Y :

min E |Y − f |2 =⇒ f =
f

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

min E |Y − f |2 =⇒ f = E (Y ) ,
f
with MSE var (Y ).

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

min E |Y − f |2 =⇒ f = E (Y ) ,
f
with MSE var (Y ).

Least squares estimate of Y , given X :

min E |Y − f (X )|2 | X =⇒ f (X ) =
f

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

min E |Y − f |2 =⇒ f = E (Y ) ,
f
with MSE var (Y ).

Least squares estimate of Y , given X :

min E |Y − f (X )|2 | X =⇒ f (X ) = E (Y |X ) ,
f
with MSE var (Y |X ).

Thus, least squares estimate of Xt+h given Xt : E (Xt+h |Xt ).

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

Conditional expectations are easy under Gaussian distributions!

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

Suppose X1 , . . . , Xn is jointly Gaussian, with density

1 1 T −1
fX (x) = n/2 1/2 exp − (x − µ) Σ (x − µ) .
2π |Σ| 2

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF

Suppose X1 , . . . , Xn is jointly Gaussian, with density

1 1 T −1
fX (x) = n/2 1/2 exp − (x − µ) Σ (x − µ) .
2π |Σ| 2
In particular, joint law of (Xt , Xt+h ) is also Gaussian, with mean
(µt , µt+h ) and covariance

σt2 ρ(t, t + h)σt σt+h
2 .
ρ(t, t + h)σt σt+h σt+h
Conditional distribution Xt+h given Xt = xt is therefore

σt+h ρ(t, t + h)(xt − µt ) 2 2
N µt+h + , σ (1 − ρ(t, t + h) ) .
σt
Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
{Xt } Gaussian and stationary. What is the optimal prediction of

Xt+h given Xt = xt ?

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
{Xt } Gaussian and stationary. What is the optimal prediction of

Xt+h given Xt = xt ?
f (xt ) = E (Xt+h |Xt = xt ) = µ + ρX (h)(xt − µ) .
The resulting MSE is

E |Xt+h − f (xt )|2 , |Xt = xt = σ 2 (1 − ρX (h)2 ) .
Prediction gets better as |ρ| increases.

f (xt ) is linear: f (xt ) = α + βt.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
For general stationary processes (not necessarily Gaussian), best

predictor (in terms of MSE) has no closed form.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
For general stationary processes (not necessarily Gaussian), best

predictor (in terms of MSE) has no closed form. However, we can
consider optimal linear predictors.

E |Xt+h − α − βXt |2 = E (α, β) .
Setting ∂α E (α, β) = 0 and ∂β E (α, β) = 0, we obtain
α = µ(1 − ρX (h)) , β = ρX (h) ,
with
MSE = σ 2 (1 − ρX (h)2 ) .

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
f (xt ) = µ + ρX (h)(xt − µ)
Optimal linear predictor for any stationary {Xt }.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
f (xt ) = µ + ρX (h)(xt − µ)

Optimal predictor for stationary and gaussian {Xt }.

Important Examples
Logistics
Stationarity
Sample ACF
Prediction with ACF
f (xt ) = µ + ρX (h)(xt − µ)

Optimal predictor for stationary and gaussian {Xt }.
Corollary: For gaussian processes, linear prediction is optimal
(in MSE).
Extension to multiple time indices.

Intro Stat 153

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Intro Stat 153

Uploaded by

Copyright:

Available Formats

Logistics

Introduction to Time Series

Stat 153: Introduction to Time Series

Joan Bruna Stat 153: Introduction to Time Series

Instructor: Joan Bruna Estrach, joan.bruna@berkeley.edu

Joan Bruna Stat 153: Introduction to Time Series

Midterm Exams: October 13th, November 12th, 15 % each.

What is a Time Series?

Joan Bruna Stat 153: Introduction to Time Series

What is a Time Series?

Joan Bruna Stat 153: Introduction to Time Series

What is a Time Series?

Statistical Model necessary.

What is a Time Series?

Interaction of random and non-random behavior.

What is a Time Series?

Joan Bruna Stat 153: Introduction to Time Series

What is a Time Series?

What is a Time Series?

Joan Bruna Stat 153: Introduction to Time Series

What is a Time Series?

Periodic, stationary phenomena is studied with spectrograms.

Joan Bruna Stat 153: Introduction to Time Series

What is a Time Series?

Control Theory (Kalman)

What is a Time Series?

Joan Bruna Stat 153: Introduction to Time Series

What is a Time Series?

State space models (language).

Joan Bruna Stat 153: Introduction to Time Series

What is a Time Series?

State space models (language).

This is not an IID world

Fundamental characteristic of time-series: in general, samples

Joan Bruna Stat 153: Introduction to Time Series

This is not an IID world

Fundamental characteristic of time-series: in general, samples

Estimating, modeling and analyzing this dependency/correlation is

Joan Bruna Stat 153: Introduction to Time Series

Objectives of Time Series Analysis

Compact description of the data: statistical modeling.

Joan Bruna Stat 153: Introduction to Time Series

Example: Monthly Sales of a Souvenir Shop

Joan Bruna Stat 153: Introduction to Time Series

Example: Trend Trend

Compact Description: Decomposition model

Joan Bruna Stat 153: Introduction to Time Series

Compact Description: Decomposition model

This model is interpretable. (eg seasonal adjustment due to

Joan Bruna Stat 153: Introduction to Time Series

Compact Description: Decomposition model

This model is interpretable. (eg seasonal adjustment due to

Joan Bruna Stat 153: Introduction to Time Series

Overview of the Course

1 Time series basics.

Joan Bruna Stat 153: Introduction to Time Series

Time Series Basics

Joan Bruna Stat 153: Introduction to Time Series

Time domain Models

Auto-Regressive models (AR, MA, ARMA, ...)

Joan Bruna Stat 153: Introduction to Time Series

Joan Bruna Stat 153: Introduction to Time Series

Joseph Fourier (1768-1830).