Stationary Stochastic Processes

Lesson 4
Umberto Triacca
Facolt`a di Economia
Universit`a dell’Aquila
umberto.triacca@gmail.com
Umberto Triacca Stationary Stochastic Processes
One-step prediction of a stochastic process
Let {x
t
} be a stochastic process.
Suppose that the variable x
t
is observed for t = 1, 2, ..., T.
We consider the problem of predicting the value x
T+1
in terms
of the values
{x
T
, ..., x
1
}
up time T
Umberto Triacca Stationary Stochastic Processes
One-step prediction of a stochastic process
More precisely, our goal is to find the linear combination of
x
T
, ..., x
1
, which forecasts x
T+1
with minimum mean square
error. This predictor, denoted by P
T
x
T+1
, has the form
P
T
x
T+1
= α
1
x
T
+ α
2
x
T−1
+ ... + α
T
x
1
where the coefficients α
1
, α
2
, ..., α
T
are determined by finding
the values that minimize the mean square error
S(α
1
, α
2
, ..., α
T
) = E(x
T+1
−α
1
x
T
−α
2
x
T−1
−... −α
T
x
1
)
2
Umberto Triacca Stationary Stochastic Processes
One-step prediction of a stochastic process
A necessary and sufficient set conditions for α
1
, α
2
, ..., α
T
to
minimize the mean square error is
E [(x
T+1
−α
1
x
T
−α
2
x
T−1
−... −α
T
x
1
)x
T+1−j
] = 0 j = 1, ..., T
Umberto Triacca Stationary Stochastic Processes
One-step prediction of a stochastic process
In vector notation we have
ΓΓΓααα = γγγ
where
ααα = (α
1
, α
2
, ..., α
T
)

Γ =





E(x
2
T
) E(x
T−1
x
T
) · · · E(x
1
x
T
)
E(x
T
x
T−1
) E(x
2
T−1
) · · · E(x
1
x
T−1
)
.
.
.
.
.
.
.
.
.
.
.
.
E(x
T
x
1
) E(x
T−1
x
1
) · · · E(x
2
1
)





and
γγγ = (E(x
T+1
x
T
), E(x
T+1
x
T−1
), ..., E(x
T+1
x
1
))

Umberto Triacca Stationary Stochastic Processes
One-step prediction of a stochastic process
Hence, the best linear predictor of x
T+1
in terms of the values
{x
T
, x
T−1
, ..., x
1
}
up time T, is given by
P
T
x
T+1
= α
1
x
T
+ α
2
x
T−1
+ ... + α
T
x
1
where the coefficients ααα = (α
1
, α
2
, ..., α
T
)

satisfies the
condition
ΓΓΓααα = γγγ
Umberto Triacca Stationary Stochastic Processes
One-step prediction of a stochastic process
Of course, since
E(x
t
x
s
) = Cov(x
t
, x
s
) + E(x
t
)E(x
s
) ∀t, s,
the predictor
P
T
x
T+1
= α
1
x
T
+ α
2
x
T−1
+ ... + α
T
x
1
can be determined only if the T + 1 means
E(x
T+1
), E(x
T
), ..., E(x
1
)
the T variances
Var(x
T
), Var(x
T−1
), ..., Var(x
1
),
and the T +
T(T−1)
2
covariances are known.
Umberto Triacca Stationary Stochastic Processes
The problem
In the practice, these parameters are unknown.
Thus they must be estimated.
T + 1 + 2T +
T(T −1)
2
TOO MANY PARAMETERS TO ESTIMATE!!!
Umberto Triacca Stationary Stochastic Processes
The problem
If T=100, we have to estimate 1376 parameters!!!
This is impossible because we only have a single realization of
the process.
Umberto Triacca Stationary Stochastic Processes
The solution
Restrictions which enable us to reduce the number of unknown
parameters involved in order to be able to deduce their values
from a single realization:
1
restrictions on the time-heterogeneity of the process;
2
restrictions on the memory of the process.
Umberto Triacca Stationary Stochastic Processes
Restricting the time-heterogeneity
Now we introduce an important class of stochastic processes
which exhibit time-homogeneity.
This is the class of stationary stochastic processes.
Umberto Triacca Stationary Stochastic Processes
Stationary Stochastic Process
Definition
Consider a stochastic process {X
t
; t ∈ Z}. Let {t
1
, t
2
, · · · , t
s
}
be a finite set of integers. The process {X
t
; t ∈ Z} is strongly
stationary if the joint distibution function of the vector
(X
t
1
+k
, X
t
2
+k
, ..., X
t
s
+k
) is equal with the one of
(X
t
1
, X
t
2
, ..., X
t
s
) for all integers s ≥ 1 and for all
t
1
, t
2
, · · · , t
s
, k ∈ Z.
Umberto Triacca Stationary Stochastic Processes
The meaning of Stationary
The meaning of stationarity is that the distribution of a
number of random variables of the stochastic process is
the same as we shift them along the time index axis.
Umberto Triacca Stationary Stochastic Processes
The meaning of Stationary
If {X
t
; t ∈ Z}is a strongly stationary process, then, for
example,
X
1
, X
2
, X
3
, ...
have the same distribution function.
(X
1
, X
3
), (X
5
, X
7
), (X
9
, X
11
), .....
have the same joint distribution function and further
(X
1
, X
3
, X
5
), (X
7
, X
9
, X
11
), (X
13
, X
15
, X
17
), ...
must have the same joint distribution function, and so on
Umberto Triacca Stationary Stochastic Processes
Example of a Discrete Stationary Stochastic
Process
Consider the discrete stochastic process
{X
t
; t ∈ N}
where X
t
= A, with A ∼ U (3, 7)
This process is of course strongly stationary.
Umberto Triacca Stationary Stochastic Processes
Example of a Discrete Non-Stationary Stochastic
Process
Consider the discrete stochastic process
{X
t
; t ∈ N}
where X
t
= tA, with A ∼ U (3, 7)
This process is not strongly stationary
Umberto Triacca Stationary Stochastic Processes
Weakly Stationary Stochastic Process
In many applications this is too strict a definition and hence
we define now stationary for a wider class of processes.
Umberto Triacca Stationary Stochastic Processes
Weakly Stationary Stochastic Process
Definition
A stochastic process {X
t
; t ∈ Z} is said to be
covariance-stationary, or weakly stationary if
1
E(X
t
) = µ ∀t
2
Var(X
t
) = γ
0
< ∞ ∀t
3
Cov(X
t
, X
t−k
) = γ
k
∀t, ∀k
Umberto Triacca Stationary Stochastic Processes
Weakly Stationary processes
A stochastic process is covariance-stationary if
1
it has the same mean value, µ, at all time points;
2
it has the same variance, γ
0
, at all time points; and
3
the covariance between the values at any two time points,
t, t −k, depend only on k, the difference between the
two times, and not on the location of the points along the
time axis.
Umberto Triacca Stationary Stochastic Processes
Weakly Stationary processes
Definition
A stochastic process {u
t
; t ∈ Z} in which the random variables
u
t
, t = 0 ±1, ±2... are such that
1
E(u
t
) = 0 ∀t
2
Var(u
t
) = σ
2
u
< ∞ ∀t
3
Cov(u
t
, u
t−k
) = 0 ∀t, ∀k
is called white noise with mean 0 and variance σ
2
u
, written
u
t
∼ WN(0, σ
2
u
)
.
If the random variables u
t
are independently and identically
distributed with mean 0 and variance σ
2
u
then we shal write
u
t
∼ IID(0, σ
2
u
)
Umberto Triacca Stationary Stochastic Processes
Weakly Stationary processes
A white noise process is weakly stationary
Umberto Triacca Stationary Stochastic Processes
White noise process that is strongly stationary
A process u
t
∼ IID(0, σ
2
u
) is a strongly stationary
This follows almost immediate from the definition.
1) Use the independence property to factor the joint
distribution into the product of the marginal distributions.
2) Then use the identical distribution property that
P(u
t
≤ b) = P(u
s
≤ b) for all t, s
Umberto Triacca Stationary Stochastic Processes
White noise process that is not strongly stationary
Let w be a random variable uniformly distributed in the
interval (0; 2π). We consider the process {Z
t
; t = 1, 2, ...}
defined by
Z
t
= cos(tw) t = 1, 2, ...
We have that
1
E(Z
t
) = 0 ∀t
2
Var(Z
t
) =
1
2
∀t
3
Cov(Z
t
, Z
t−k
) = 0 ∀t, ∀k
Thus Z
t
∼ WN(0, .5). However, it can be shown that is not
strongly stationary.
Umberto Triacca Stationary Stochastic Processes
Example of a Discrete Non-Stationary Stochastic
Process
Consider the discrete stochastic process
{Y
t
; t ∈ Z}
where Y
t
= α + βt + u
t
, with u
t
∼ WN(0, σ
2
u
)
This process is not weakly stationary
Umberto Triacca Stationary Stochastic Processes
Example of a Discrete Non-Stationary Stochastic
Process
A possible realization of the stochastic process
Y
t
= 3 + 7t + u
t
, with u
t
∼ WN(0, σ
2
u
)
Umberto Triacca Stationary Stochastic Processes
Example of a Discrete Non-Stationary Stochastic
Process
Consider the discrete stochastic process
{y
t
; t = 0, 1, 2, ...}
where y
0
= δ < ∞ and y
t
= y
t−1
+ u
t
for t = 1,2,..., with
u
t
∼ WN(0, σ
2
u
)
This process is called random walk.
Umberto Triacca Stationary Stochastic Processes
Example of a Discrete Non-Stationary Stochastic
Process
The mean and the variance of y
t
are be given by
E(y
t
) = δ and Var(y
t
) = tσ
2
u
Thus a random walk is not weakly stationary process.
Umberto Triacca Stationary Stochastic Processes
Example of a Discrete Non-Stationary Stochastic
Process
A possible realization of a random walk
Umberto Triacca Stationary Stochastic Processes
Example of a Discrete Covariance-Stationary
Stochastic Process
Consider the discrete stochastic process
{y
t
; t ∈ Z}
where and y
t
= u
t
+ 3u
t−1
, with u
t
∼ WN(0, σ
2
u
)
This process is covariance-stationary.
Umberto Triacca Stationary Stochastic Processes
Example of a Discrete Non-Stationary Stochastic
Process
A possible realization of a this process is given by
Umberto Triacca Stationary Stochastic Processes
Relation between strong and weak Stationarity
If the process {X
t
; t ∈ Z} is strongly stationary and has finite
second moments, then {X
t
; t ∈ Z} is weakly stationary.
Proof: Consider a strongly stationary process, {X
t
; t ∈ Z},
with finite second moments. By definition of strongly
stationary process follows that X
t
has the same distribution for
each t ∈ Z. Since the process {X
t
; t ∈ Z} has finite second
moment, this implies that E(X
t
) and Var(X
t
) are both
constant. Further X
t
and X
t−k
has the same joint distribution
and hence the same covariance for all k and for all t. Thus a
strongly stationary stochastic process with finite second
moments is weakly stationary.
Umberto Triacca Stationary Stochastic Processes
Relation between strong and weak Stationarity
Of course a weakly stationary process is not necessarily
strongly stationary.
There is one important case however in which weak
stationarity does imply strong stationarity.
Umberto Triacca Stationary Stochastic Processes
Relation between strong and weak Stationarity
If {X
t
; t ∈ Z} is a weakly stationary Gaussian stochastic
process, then {X
t
; t ∈ Z} is strongly stationary
Umberto Triacca Stationary Stochastic Processes