Time Series Analysis

What is Time Series?

• A Time series is a set of observations, each one being
recorded at a specific time. (Annual GDP of a country, Sales
figure, etc)
• A discrete time series is one in which the set of time points at
which observations are made is a discrete set. (All above
including irregularly spaced data)
• Continuous time series are obtained when observations are
made continuously over some time intervals. It is a
theoretical Concept. (Roughly, ECG graph).
• A discrete valued time series is one which takes discrete
values. (No of accidents, No of transaction etc.).
 3

Few Time series Plots

Annual GDP of USA
 4

A discrete time series is one in which the setof time points at which
observations are made is a discrete set. (All above including
irregularly spaced data)
 5

Continuous time series are obtained when observations are made

continuously over some time intervals. (ECG graph).
 6

A discrete valued time series is one which takes discrete values.

(No of accidents, No of transaction etc.).
Time series plot on car accident in U.K.
 -0.4
-0.3
-0.2
-0.1

0.1
0.2
0.3
0.4
0.5
0
1801
1808
1815
1822

Continuous time series data (Stock returns):

1829
1836
1843
1850
1857
1864
1871

Stock Returns
1878
1885
Stock Returns

1892
1899
1906
1913
1920
1927
1934
1941
1948
1955
1962
1969
1976
1983
1990
1997
2004
7
 100
150
200
250
300
350
400
50
0
1/1/1950

Time series data (Number of sunspots) showing cycles:

1/1/1953
1/1/1956
1/1/1959
1/1/1962
1/1/1965
1/1/1968
1/1/1971
1/1/1974

Sunspots
1/1/1977
Sunspots

1/1/1980
1/1/1983
1/1/1986
1/1/1989
1/1/1992
1/1/1995
1/1/1998
1/1/2001
1/1/2004
1/1/2007
1/1/2010
8
 9

Quarterly Sales of Ice-cream

Q1-Dec-Jan
 10

Objective of Time Series Analysis

• Forecasting (Knowing future is our innate wish).

• Control (whether anything is going wrong, think of ECG,
production process etc)
• Understanding feature of the data including seasonality, cycle,
trend and its nature. Degree of seasonality in agricultural price
may indicate degree of development. Trend and cycle may
mislead each other (Global temperature may be an interesting
case)
 11

Objective

• Description: Plot the data. Try to feel the data.

Some descriptive statistics may be calculated to
get some ideas about the data.
• Explanation: Deeper understanding of the
mechanism that generated the time series.
 12

Stochastic processes Approach

• Time series are an example of a stochastic or random

process
• A stochastic process is a statistical phenomenon that
evolves in time according to probabilistic laws.
• Mathematically, a stochastic process is an indexed
collection of random variables

{𝑦𝑡: 𝑡 ∈ 𝑇}
 13

Stochastic processes
• We are concerned only with processes
indexed by time, either discrete time or
continuous time processes such as
{𝑦𝑡: 𝑡 ∈ (−∞, ∞)} = {𝑦𝑡: −∞ < 𝑡 < ∞}
Or
{𝑦𝑡: 𝑡 ∈ (1,2,3, … )} = {𝑦1,𝑦2,𝑦3, … }
 14

Stochastic Process
• A stochastic process{𝑦𝑡}∞𝑡=−∞ is a collection
of random variables or a process that
develops in time according to probabilistic
laws.
• The theory of stochastic processes gives us a
formal way to look at time series variables.
 15

DEFINITION
𝑦(𝑤, 𝑡): 𝑠𝑡𝑜𝑐ℎ𝑎𝑠𝑡i𝑐 𝑝𝑟𝑜𝑐𝑒𝑠𝑠

Sample Space Index Set

• For a fixed t, y(w, t) is a random variable.

• For a given w, y(w, t) is called a sample function or a
realization as a function of 𝑡.
 16

Stochastic Process
• Time series is a realization or sample function from a certain
stochastic process.
• A time series is a set of observations generated sequentially in
time. Therefore, they are dependent to each other. This means
that we do NOT have random sample.
• We assume that observations are equally spaced in time.

• We also assume that closer observations might have stronger

dependency.
 17

JOINT PDF OF A TIME SERIES

• Remember that

𝐹𝑦1 (𝑦1): 𝑡ℎ𝑒 𝑚𝑎𝑟𝑔i𝑛𝑎𝑙 𝑐𝑑ƒ

ƒ𝑦1 (𝑦1): 𝑡ℎ𝑒 𝑚𝑎𝑟𝑔i𝑛𝑎𝑙 𝑝𝑑ƒ

𝐹𝑦1 ,𝑦2 ,…,𝑦𝑛 (𝑦1 , 𝑦2 , … , 𝑦𝑛 ): 𝑡ℎ𝑒 j𝑜i𝑛𝑡 𝑐𝑑ƒ

ƒ𝑦1 ,𝑦2 ,…,𝑦𝑛 (𝑦1 , 𝑦2 , … , 𝑦𝑛 ): 𝑡ℎ𝑒 j𝑜i𝑛𝑡 𝑝𝑑ƒ

 18

JOINT PDF OF A TIME SERIES

• For the observed time series, say we have two
points, 𝑡 and 𝑠.
• The marginal pdfs: ƒF𝑡(𝑦𝑡)𝑎𝑛𝑑 ƒF𝑠 (𝑦𝑠)

• The joint pdf: ƒF𝑡,F𝑠 (𝑦𝑡, 𝑦𝑠) G ƒF𝑡(𝑦𝑡). ƒF𝑠 (𝑦𝑠)

 19

JOINT PDF OF A TIME SERIES

• Since we have only one observation for each r.v.
𝑌𝑡, inference is too complicated if distributions
(or moments) change for all 𝑡 (i.e. change over
time). So, we need a simplification.
15

Y4
10
Y6 r.v.
5

0
1 2 3 4 5 6 7 8 9 10 11 12
 20

JOINT PDF OF A TIME SERIES

• To be able to identify the structure of the series,
we need the joint pdf of 𝑦1, 𝑦2, … , 𝑦𝑇. However,
we have only one sample (realization). That is,
one observation from each random variable.
• This is in complete contrast to that of a cross-
section/survey data. For cross section data, for a
given population, we have a random sample.
Based on the sample we try to infer about the
population.
 21

JOINT PDF OF A TIME SERIES

• In Time series, each random variable has one
distribution/population. And from each
population we have just one observation. So
inference is not feasible unless we have some
strong restrictive assumptions.
• Therefore, it is very difficult to identify the joint
distribution. Hence, we need an assumption to
simplify our problem. This simplifying
assumption is known as STATIONARITY.
 22

STATIONARITY
• The most vital and common assumption in time
series analysis.
• The basic idea of stationarity is that the
probability laws governing the process do not
change with time.
• The process is in statistical equilibrium.
 23

Why does Stationarity Assumption work?

• Now, suppose each distribution has same mean. In that
case the common mean could be estimated based on
the realization of size ‘n’.
• We can visualize the fact in the following way---
Suppose we have 10 identical machines producing some
item, say, bulb. Suppose each machine is run for one hour.
Now it is easy to visualize that total (average) output by 10
machines is same as that of total (average) output by a
single machine running for 10 hours.
 24

TYPES OF STATIONARITY
• STRICT (STRONG OR COMPLETE) STATIONARY
PROCESS: Consider a finite set of r.v.s.
𝑌𝑡1 , 𝑌𝑡2 , … , 𝑌𝑡𝑛 from a stochastic process {𝑌(𝑤, 𝑡); 𝑡 =
0, ±1, ±2, … }.

• The n-dimensional distribution function is defined by

𝐹F𝑡1,F𝑡2,…,F𝑡𝑛 (𝑦𝑡1 , 𝑦𝑡2 , … , 𝑦𝑡𝑛) = 𝑃(𝑤: 𝑌𝑡1 < 𝑦1, … , 𝑌𝑡1 < 𝑦𝑛)

where 𝑦i, i = 1,2, … , 𝑛 are any real numbers.

 25

STRONG STATIONARITY
• A process is said to be first order stationary in distribution,
if its one dimensional distribution function is time-invariant,
i.e., 𝐹F𝑡 (𝑦1) = 𝐹F𝑡 (𝑦1) for any 𝑡1 and 𝑘.
1 1+𝑘

• Second order stationary in distribution if 𝐹F𝑡1,F𝑡2 (𝑦1, 𝑦2) =

𝐹F𝑡 ,F (𝑦1, 𝑦2) for any 𝑡1, 𝑡2 and 𝑘.
1+𝑘 𝑡2+𝑘

• nth order stationary in distribution if

𝐹F𝑡1,F𝑡2,…,F𝑡𝑛 (𝑦1, 𝑦2, … , 𝑦𝑛) = 𝐹F𝑡 ,F ,…,F𝑡 (𝑦1, 𝑦2, … , 𝑦𝑛)
1+𝑘 𝑡2+𝑘 𝑛+𝑘
for any 𝑡1, … , 𝑡𝑛 and 𝑘.
 26

STRONG STATIONARITY
𝑛𝑡ℎ order stationarity in distribution = strong
stationarity
→ Shifting the time origin by an amount “𝑘” has
no effect on the joint distribution, which must
therefore depend only on time intervals between
𝑡1, 𝑡2 … , 𝑡𝑛 not on absolute time, 𝑡.
 27

STRONG STATIONARITY
• So, for a strong stationary process
i. ƒF𝑡1 ,F𝑡2 ,…,F𝑡𝑛 (𝑦1, 𝑦2, … , 𝑦𝑛) = ƒF𝑡1+𝑘,F𝑡2+𝑘,…,F𝑡𝑛+𝑘 (𝑦1, 𝑦2, … , 𝑦𝑛)
ii. 𝐸(𝑌𝑡) = 𝐸(𝑌𝑡+𝑘) ⇒ 𝜇𝑡 = 𝜇𝑡+𝑘 = 𝜇 ✯𝑡, 𝑘

Expected value of a series is constant over time, not a function of time

iii. 𝑉𝑎𝑟(𝑌𝑡) = 𝑉𝑎𝑟(𝑌𝑡+𝑘) ⇒ 𝜎2 = 𝜎2 = 𝜎2 ✯𝑡, 𝑘
𝑡 𝑡+𝑘
The variance of a series is constant over time, homoscedastic
iv. 𝑐𝑜𝑣(𝑌𝑡, 𝑌𝑠) = 𝑐𝑜𝑣(𝑌𝑡+𝑘, 𝑌𝑠+𝑘) ⇒ 𝛾𝑡,𝑠 = 𝛾𝑡+𝑘,𝑠+𝑘, ✯𝑡, 𝑘 ⇒
𝛾|𝑡−𝑠| = 𝛾|𝑡+𝑘−𝑠−𝑘| = 𝛾ℎ

Not constant, not depend on time, depends on time interval, which we

call “lag”, 𝑘.
 28

STRONG STATIONARITY

20

10

0
1 2 3 4 5 6 7 8 9 10 11 12

𝑐𝑜𝑣(𝑦2𝑦1) = 𝛾2−1 = 𝛾1

𝑐𝑜𝑣(𝑦3𝑦2) = 𝛾3−2 = 𝛾1

𝑐𝑜𝑣(𝑦𝑛𝑦𝑛−1) = 𝛾𝑛−(𝑛−1) = 𝛾1 Affected from time lag, 𝑘.

𝑐𝑜𝑣(𝑦3𝑦1) = 𝛾3−1 = 𝛾2

𝑐𝑜𝑣(𝑦1𝑦3) = 𝛾1−3 = 𝛾−2

 29

STRONG STATIONARITY
v. 𝑐𝑜𝑟𝑟[𝑦𝑡𝑦𝑠] = 𝑐𝑜𝑟𝑟[𝑦𝑡+𝑘𝑦𝑠+𝑘] ⇒ 𝜌𝑡,𝑠 = 𝜌𝑡+𝑘,𝑠+𝑘 ✯𝑡,𝑘
⇒ 𝜌|𝑡−𝑠| = 𝜌|𝑡+𝑘−𝑠−𝑘| = 𝜌ℎ

Let 𝑡 = 𝑡 − 𝑘 and 𝑠 = 𝑡,

𝜌𝑡,𝑡−𝑘 = 𝜌𝑡+𝑘,𝑡 = 𝜌𝑘 ✯𝑡, 𝑘

Remark: We have assumed the existence of 2nd order moments.

• It is usually impossible to verify a distribution particularly a

joint distribution function from an observed time series. So,
we use weaker sense of stationarity.
 30

WEAK STATIONARITY
• WEAK (COVARIANCE) STATIONARITY OR
STATIONARITY IN WIDE SENSE: A time series is
said to be covariance stationary if its first and second
order moments are unaffected by a change of time
origin.
• That is, we have constant mean and variance with
covariance and correlation beings functions of the time
difference only.
 31

WEAK STATIONARITY

𝐸[𝑦𝑡] = 𝜇, ✯𝑡

𝑣𝑎𝑟[𝑦𝑡] = 𝜎 2 < ∞, ✯𝑡
𝑐𝑜𝑣[𝑦𝑡, 𝑦𝑡−𝑘] = 𝛾𝑘, ✯𝑡

𝑐𝑜𝑟𝑟[𝑦𝑡, 𝑦𝑡−𝑘] = 𝜌𝑘, ✯𝑡

From, now on, when we say “stationary”, we imply weak
stationarity.
 32

EXAMPLE

• Consider a time series {Yt} where

𝑌𝑡 = 𝑒𝑡
and 𝑒𝑡∼ii𝑑(0, 𝜎2). Is the process stationary?
 33

EXAMPLE

• MOVING AVERAGE: Suppose that {𝑌𝑡} is

constructed as
𝑒𝑡 + 𝑒𝑡−1
𝑦𝑡 =
2
And 𝑒𝑡∼ii𝑑(0, 𝜎2). Is the process {𝑦𝑡} stationary?
 34

EXAMPLE
• RANDOM WALK

𝑦𝑡 = 𝑒1 + 𝑒2 + ⋯ + 𝑒𝑡

where 𝑒𝑡∼ii𝑑(0, 𝜎2).. Is the process {𝑦𝑡} stationary?

 35

EXAMPLE
• Suppose that time series has the form
𝑦𝑡 = 𝑎 + 𝑏𝑡 + 𝑒𝑡
where 𝑎 and 𝑏 are constants and 𝑒𝑡 is a weakly stationary
process with mean 0 and autocovariance function 𝛾𝑘. Is {𝑦𝑡}
stationary?
 36

EXAMPLE

𝑦𝑡 = (−1)𝑡𝑒𝑡
where 𝑒𝑡∼ii𝑑(0, 𝜎2).. Is the process {𝑦𝑡} stationary?
 37

STRONG VERSUS WEAK STATIONARITY

• Strict stationarity means that the joint distribution only depends on the
‘difference’ h, not the time (t1, . . . , tk).

• Finite variance is not assumed in the definition of strong stationarity,

therefore, strict stationarity does not necessarily imply weak
stationarity. For example, processes like i.i.d. Cauchy is strictly
stationary but not weak stationary.

• A nonlinear function of a strict stationary variable is still strictly

stationary, but this is not true for weak stationary. For example, the
square of a covariance stationary process may not have finite variance.

• Weak stationarity usually does not imply strict stationarity as higher

moments of the process may depend on time t.
 38

STRONG VERSUS WEAK STATIONARITY

• If process {𝑦𝑡} is a Gaussian time series, which means

that the distribution functions of {𝑦𝑡} are all
multivariate Normal, weak stationary also implies
strict stationary. This is because a multivariate Normal
distribution is fully characterized by its first two
moments.
 39

STRONG VERSUS WEAK STATIONARITY

• For example, a white noise is stationary but may not be
strict stationary, but a Gaussian white noise is strict
stationary. Also, general white noise only implies
uncorrelation while Gaussian white noise also implies
independence. Because if a process is Gaussian,
uncorrelation implies independence. Therefore, a
Gaussian white noise is just ii𝑑 𝑁(0, 𝜎2).
 40

Measure of Dependence--Autocovariance
• Because the random variables comprising the process
are not independent, we must also specify their
covariance

𝛾𝑡1,𝑡2 = 𝑐𝑜𝑣(𝑦𝑡1 , 𝑦𝑡2 )

41