Topic 4 :

STOCHASTIC TIME
SERIES

1
STOCHASTIC TIME SERIES

1. STOCHASTIC TIME SERIES

2. STATIONARY TIME SERIES
3. NON-STATIONARY TIME SERIES
4. STATIONARITY TESTING
5. FORECASTING UNDER STATIONARITY CONDITION
6. FORECASTING UNDER NON-STATIONARITY CONDITION
Definition

• Sequence of random variables;

𝑌𝑡 : 𝑡 = 0, ±1, ±2, … .
• is called a stochastic process and serves as a model for an observed
time series.
• the complete probabilistic structure of such process is determined
by the set of distribution of all collections of the Y’s.
Definition

• An observed time series is considered to be one realization of a stochastic

process.
• A set of observed time series is considered to be a sample of the
population.
𝑌1 = 𝑦1 , 𝑌2 = 𝑦2 , … , 𝑌𝑇 = 𝑦𝑇 = 𝑦𝑡 𝑇𝑡=1

• Goal of time series modelling is to describe the probabilistic behaviour of

the underlying stochastic process that is believed to have generated the
observed data in a concise way.
• We want to be able to use the observed sample data to estimate
important characteristics of a time series model such as measure of time
dependence.
Statistical Properties

• Mean function is defined as;

is the expected value of the process of the process at time- can be different at each
time point

• Variance function is defined as;

Statistical Properties

• Autocovariance function is defined as;

for 𝑡 = 0, ±1, ±2, … .

• Autocovariance in a given stochastic process is a function that gives the

covariance (measure of the joint variability of two random variables), of the
process with itself at pairs of time points, t and s.
Statistical Properties
• The autocorrelation function is defined as;

for

where

• The autocorrelation, also known as serial correlation is the correlation between the elements of
a series and others from the same series separated from them by given times, t and s.
• Values of 𝜌𝑡,𝑠 near ±1 indicate strong (linear) dependence, whereas values near zero indicate
weak (linear) dependence.
• if 𝜌𝑡,𝑠 = 0, we say that 𝑌𝑡 and 𝑌𝑠 are uncorrelated.
STATIONARY TIME SERIES
Stationary Time Series
• To make statistical inference about the structure of a stochastic process on the
basis of an observed record of that process, we must make some simplifying (and
presumably reasonable) assumption about the structure.

• Basic idea of stationary is that the probability laws that govern the behaviour of
the stochastic process do not change over time.

• Two types of stationarity statistical properties;

• strictly stationary (or strongly stationary).
• weakly stationary (or second-order stationary).
Strictly Stationary Time Series
• A process 𝑌𝑡 is said to be strictly stationary if;
• The joint distribution of 𝑌𝑡1 , 𝑌𝑡2 ,…, 𝑌𝑡𝑛 is the same as the joint distribution of
𝑌𝑡1−𝑘 , 𝑌𝑡2−𝑘 ,…, 𝑌𝑡𝑛−𝑘 for all choices of time points 𝑡1 , 𝑡2 , … , 𝑡𝑛 and all choices of time lag k

• When n=1 (univariate) the distribution of 𝑌𝑡 is the same as that of 𝑌𝑡−𝑘 for all t and k
in other words, Y’s are (marginally) identically distributed.
• 𝐸 𝑌𝑡 = 𝐸 𝑌𝑡−𝑘 for all t and k so that the mean function is constant for all times.
• 𝑉𝑎𝑟 𝑌𝑡 = 𝑉𝑎 𝑌𝑡−𝑘 for all t and k so that the variance is also constant over time.
Strictly Stationary Time Series
When n=2?
Strictly Stationary Time Series
When n=2?
•Thus, for a stationary process, we simplify the notation and write

and

•In most practical applications and lack of perfection, typically it is virtually

impossible to test for strict stationarity.
Weakly Stationary Time Series
• A process 𝑌𝑡 is said to be weakly stationary (second order stationary) if;

1. The mean function is constant over time and finite;

2. The variance function is constant and finite;

3. The covariance between 𝑌𝑡 and 𝑌𝑡−𝑘 is independent of t and only differentiated by

the distance in time, k ,

for all time t and lag k

4.The corresponding 𝑘𝑡ℎ order autocorrelation is given by;

• A stationary time series is characterised by its time-invariant (or constant)

mean, variance and auto-covariances.
• In this course, the term stationary will be referred to weakly stationary.
Example

• Constant mean and variance.

• Constant autocovariance can be detected using some other measure.
White Noise Process

• Important example of stationary process is so-called white noise process in

a mathematical form of;

where the parameter 𝜃0 is a known constant term in the model;

• We said 𝜀𝑡 is the error term, randomly generated from a population and
assumed to be independent and normally distributed with mean zero and
constant variance
• In short, the white noise error term can be presented as;
White Noise Process
1.By having zero mean;
• Taking the expectation on both sides of the white noise process,

since by definition 𝐸 𝜀𝑡 = 0.
Thus, the random errors has no effect on the expectation value of the variable 𝑌𝑡 .

2.By having a constant variance, (homocedasticity);

3.Each random errors are unrelated (independence)

4.Random errors are generated from the same population (identical)

White Noise Process

• Based on the analogy of white light and indicates that all possible periodic
oscillations are present with equal strength.
• A special case of stationary time series.
• The most “clean or fundamental” time series, thus many useful processes can be
constructed from white noise.
• A model is said to be a good fit if its residual error series appear to be a realisation of
independent series.
 White Noise vs Stationary Time Series

• Visually hard to distinguish.

• Statistical test can be used to
distinguish one from the other.
NON-STATIONARY TIME
SERIES
Definition

• Does not satisfy the stationarity conditions as described previously

• Can be categorised into two classes;
1. A series that is not constant around the mean or level (due to trend or
seasonal pattern) and hence can be expressed as deterministic
function of time.
2. A series that shows an increase in the variability of the observations
through time (in addition to time dependent)
Example

• Structural changes over time;

• Left Figure: Changes on level
• Right Figure: Changes on variability
• Non-stationary time series data can be made stationary. How?
• A simple procedure used to remove the non-stationarity in time series is to
perform the differencing which will be described detailed in the next topic.
• A series that requires first difference to be stationary is said to be
integrated of order one or first order integrated series.
• If a series does not require differencing, it is said to be integrated of order
zero.
Non-Stationary Time Series-Random Walk Process

• Let 𝜀1 , 𝜀2 , … be a sequence of independent, identically distributed random

variables each with zero mean and constant variance.
• The observed time series, 𝑌𝑡 : 𝑡 = 1,2, … is constructed as follows;
Random Walk Process

• Alternatively, we can write:

with initial condition 𝑌1 = 𝜀1

• If the 𝜀’s are interpreted as the sizes of the “steps” taken (forward or
backward) along the number line, then 𝑌𝑡 is the position of the “random
walker” at time t.
Random Walk Process

1.Mean function for a random walk process;

• so that 𝜇𝑡 = 0 for all t.

Random Walk Process

2.For variance,

• The variance increases linearly with time.

• Note that even though the theoretical mean function is zero for all time
points, the fact that the variance increases over time and the correlation
between process values nearby in time is nearly 1 indicate that we should
expect long excursions of the process away from the mean level of zero.
Random Walk with Drift Process

• The process;

is defined as the random walk with drift since it fluctuates around a constant mean, 𝜇.
• Can be shown by the process of substitution that;

• Therefore if the observed time series is a realization of a random walk model, it can
be shown that forecasting for k period ahead is simply;
Random Walk with Drift Process

•To show the realisation of this model is basically its expected value,

since
Example

• Black line (random walk with drift), blue line (random walk without drift).
TESTING FOR STATIONARITY
Autocorrelation Function (ACF)-correlogram

• The auto-covariance of lag k;

• The auto-correlation of lag k;
measures the correlation between 𝑌𝑡 and 𝑌𝑡−𝑘

• Plotting of auto-correlation coefficient against k yields Autocorrelation Function (ACF)

• ACF summarized the dependence structure of a time series.
• Time series with different statistical properties (white noise, stationary, trend,
seasonality, etc) show different patterns in ACF.
Example: White Noise ACF plot

POPULATION ACF
Sample Autocorrelation Function (SACF)

• 𝜌’s are unknown and should be estimated from observed time series data.
• Sample autocorrelation of lag k;

• Plot of 𝜌’s
ො against k is called sample ACF (SACF).
• SACF measures dependence structure of a time series.
Statistical Significance of SACF-individual test

• Testing the hypothesis based on the confidence band of 95% confidence limit of the
standard error of autocorrelation.
• The hypothesis:

• Reject the null hypothesis at 5% level of significance if the sample value 𝜌ො𝑘 lies
outside the confidence band ±1.96𝑇 −1/2 where T is the number of observations in
the series.
Statistical Significance of SACF-individual test
Example

• The first 5 autocorrelations coefficient using a series of length 100

observations to be;
• A coefficient would be significant if it lies outside (-0.196, 0.196) at the 5%
level of significance.
• None of the autocorrelation is significant, accept the null hypothesis of
white noise.
Statistical Significance of SACF-joint test (Ljung-Box test)
• Hypothesis:

• Test-statistics:

• Critical Value:

• Decision:
Reject the null hypothesis if

• A reasonably large value should be chosen for m

Example
• The first 5 autocorrelations coefficient using a series of length 100 observations to
be; 0.144, 0.091, -0.158, -0.152, 0.087.
• Hypothesis? Test statistic? Critical value? Decision?

• Compared to a tabulated 𝜒52 =11.1 at the 5% level of significance,

• Thus, we fail to reject the null hypothesis of joint autocorrelations are equal to zero
(white noise).
Statistical Significance of SACF-summary

• When testing for a white noise using SACF;

• A white noise has no dependence on its own past.
• 𝜌𝑘 = 0 for all k for a white noise process.
• Evidence for white noise;
• If SACF shows no values outside the 95% confidence band for
individual test.
• If 𝐻0 of Ljung-Box test is not rejected for a sufficiently large value of
m.
SACF of non-white noise series-AR(1)

• An AR(1) process can be modelled as:

• An AR(1) model is stationary if

• The process will be called as a random walk process (non-stationary) if 𝛼 = 1
• Successive Y’s are dependent on its past for non-stationary series.
• The degree of dependence depends on the value of 𝛼.
• White noise is a special case of AR(1) model if 𝛼 = 0 .
Example: AR(1) Simulated data with different 𝜶
Example: SACF of a Simulated AR(1)
SACF of non-white noise series

• Sample autocorrelation function shows a decline.

• The speed of decline depend on the degree of dependence on past.
• At least one 𝜌ො𝑘 is outside the 95% confidence band.
• Ljung-Box test statistics shows rejection for a reasonably large value of m .
Stationarity Checking

By using SACF
• A stationary time series shows SACF that quickly dies out - only first a few 𝜌ො𝑘 ‘s are
not equal to zero statistically.
• However, if SACF shows flat decline (slowly decaying), it is an evidence of possible
non-stationary.

By using Time Plot

• If time plot shows smooth movement showing global trend or a few local trends, it is
evidence of possible non-stationary.
• If time plot shows frequent up and downs with no noticeable trend, it is evidence of
stationary.
• These measure are descriptive to a large extent, and can be subjective and
inaccurate in checking stationarity.
• We will use statistical inference called the unit root testing (next topic).
 FORECASTING UNDER
STATIONARIY CONDITION
VS
NON-STATIONARITY CONDITIONS
 Forecasting under stationarity condition-one step ahead
forecast
•We consider the equation in which 𝑦𝑡 is stationary about 𝜃0 .
•Hence the one step ahead forecast;

•By using the conditional expectation procedure (where the forecast is conditional on its
past value) we can show that the forecast for period T+1 is basically 𝑦ො𝑇+1 = 𝜃0 .
•Since all 𝑦𝑡 , 𝑡 = 0, ±1, ±2, …,T are independent, the probability of distribution is not
affected by whatever given values of 𝑦1 , 𝑦2 , … , so that the expected value of 𝑦𝑇+1 ;

since
• One-step-ahead forecast error is basically a random shock 𝑒𝑇+1 ;

𝑒𝑇 is used instead 𝜀𝑇 to differentiate the actual calculated error terms against its
corresponding theoretical value, 𝜀𝑇 .

• Hence for any white noise model, the forecast error for lead time m is;

and follows the variance of the forecast errors is given as;

Forecasting under non-stationarity condition

•Let’s us consider the random walk process with drift as example.

•For this type of series, it can be shown that the forecast values increase or decrease linearly with k.
• The variance of 𝑦𝑇+𝑘 increases as the value of k increases
• Values that neither converge nor stabilize
• Large values of the sample autocorrelation (≈ 1) for all time periods

• The variance of forecast errors under this condition increases as t increases

• It follows that the widths of the prediction intervals also increase or decrease
with the square root of k
References

Mohd. Alias Lazim. (2007). Introductory Business Forecasting a practical

approach. University Publication Centre (UPENA).

Cryer, J. D., & Chan, K. S. (2011). Time Series Analysis With Applications
in R . Springer.