WMS

ST202
Stochastic Processes

Revision Guide

Written by Roseanna Ferguson

WMS
ii ST202 Stochastic Processes

Contents
1 Discrete Time Markov Chains 1
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Transition Matrices P . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

2 Classification of States: Communicating states, periodicity, recurrence and tran-

sience. 1
2.1 Communicating Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.2 Periodicity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.3 Recurrence or Transience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

3 The Fundamental Matrix G 2

4 Applications 2
4.1 Genetics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
4.2 Branching Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
4.3 Random Walks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

5 Renewal Theory 3
5.1 Renewal Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
5.2 Delayed Renewal Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
5.3 Stationary Renewal Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
5.4 Coupling and the Renewal Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

6 Limiting Behaviour 3

7 The Gibbs Sampler 3

Introduction
Stochastic Processes involves a few proofs but mostly it is about understanding the theory sufficiently
to be able to apply it in calculations. This guide will include examples but make sure you understand
thoroughly the examples that have been included in the lectures because questions in the exam are likely
to be similar to these. Redo the assignments as extra practice and recommended books on the module
web page will provide even more practice material.
An excellent technique of preparing for the exam is simply to read through the revision guide until you
come to the first example or proof, cover it up, and try and answer it without looking. If you get stuck,
then (and only then) you should look at the first step in the example or proof, take it as a hint and try
and answer the rest; then repeat this procedure until you have worked through the entire guide.

Disclaimer: Use at your own risk. No guarantee is made that this revision guide is accurate or
complete, or that it will improve your exam performance. Use of this guide will increase entropy,
contributing to the heat death of the universe. Contains no GM ingredients. Your mileage may vary.
All your base are belong to us.

Authors
This revision guide for ST202 Stochastic Processes has been designed as an aid to revision, not a
substitute for it. Use at your own risk. Written by Roseanna Ferguson.
Based upon lectures given by Dr.Krzysztof Latuszyski at the University of Warwick, 2013–2014.
Any corrections or improvements should be entered into our feedback form at http://tinyurl.com/WMSGuides
(alternatively email revision.guides@warwickmaths.org).
ST202 Stochastic Processes 1

1 Discrete Time Markov Chains

1.1 Background
So Firstly, let us define some of the basics

Definition 1.1 (Stochastic process). A stochastic process X is a family {Xt , t ∈ J} of random variables
indexed by a time set J (which is usually non-negative integers).

Definition 1.2. The state space S of a stochastic process X is the set of all values Xn can take. An
event is a subspace of the state space

For example....

Definition 1.3 (Discrete Markov Chain). X = {Xn : n = 0, 1, . . . } is said to be a discrete state space
Markov chain if:
1. The state space, S is discrete and;
2. The Markov property holds.

The Markov property is a key concept in Stochastic Processes...The intuition is P[f uture | present, past] =
P[f uture | present]. Let us define it rigorously.

Definition 1.4 (The Markov property). The conditional probabilities of the future given the past depend
only on the present.

P[Xn+1 = xn+1 | Xn = xn , Xn−1 = xn−1 , . . . , X0 = x0 ] = P[Xn+1 = xn+1 | Xn = xn ]

1.2 Transition Matrices P

A transition matrix stores the transition probabilities of a discrete markov state space. A transition
probability, denoted Pij is the probability of moving from one state i to another state j.

Pij = P[Xn+1 = j | Xn = i]

In this module we will mostly deal with stationary transition probabilities, so assume that any transition
probabilities are stationary henceforth.

Definition 1.5 (Stationary transition probability). The discrete Markov state space is said to have
stationary transition probabilities if whenever the transition probabilities, Pij = P[Xn+1 = j | Xn = i],
are well defined they do not vary with time n.

Definition 1.6 (Transition probability matrix). 1. The one-step transition probability matrix is de-
noted by P = [P (i, j)]i,j∈s where P (i, j) = Pij = P[Xm+1 = j | Xm = i] the one-step transition
probabilities.
2. The n-step transition probability matrix is denoted by P (n) = [P n (i, j)]i,j∈s where P n (i, j) = Pijn =
P[Xm+n = j | Xm = i] the n-step transition probabilities that is the probability we get to state j
after n steps when starting at state i.
3. P (0) = I|S|X|S| I.e the identity matrix which is the size of the state space S.

2 Classification of States: Communicating states, periodicity,

recurrence and transience.
Definition 2.1 (State diagram). A state diagram is a directed graph where vertices are states and the
directed edges are possible one-step transitions.
2 ST202 Stochastic Processes

2.1 Communicating Classes

Definition 2.2 (i leads to j). Let i, j ∈ S. i leads to j, denoted i → j, if ∃n > 0 such that Pijn > 0.

This means the probability of going to j if we started from i is greater than zero.

Definition 2.3 (i, j intercommunicate). Let i, j ∈ S. i, j intercommunicate, denoted ilef trightarrowj

if i → j and j → i.

This means it is possible to go from i to j and from j to i.

Definition 2.4 (Communicating class). Let i, j ∈ S. Then i, j are in the same communicating class,
denoted i:j, if either i ↔ j or i = j.

So two states are in the same communicating class if we can get from one to the other and back
again. Note that a stochastic matrix P is irreducible if all states intercommunicate i.e the state space,
S, is one single communicating class.

Definition 2.5 (i is essential). Let i ∈ S. Then i is essential if ∀j ∈ S i ↔ j ⇒ j → i.

2.2 Periodicity
Definition 2.6. The period of a state is the greatest common divisor (gcd) of possible return times.
(n)
1. If i → i then P eriod(i) = gcd{n > 0 : Pii > 0}
2. If it is not the case that i → i then the period(i) is undefined.
3. If period(i) = 1 then it is aperiodic.

2.3 Recurrence or Transience

First-step decomposition. First-passage decomposition.

Definition 2.7 (Recurrence and Transience). The state j is

(∗)
1. Recurrent if fjj = 1
(∗)
2. Transient if fjj < 1

So recurrence means it is certain we return to the state j if we start at j. Transience is that it could
be the case that you never return to state j.

3 The Fundamental Matrix G

Definition 3.1 (The fundamental matrix G). The fundamental matrix of a substochastic matrix Q is
given by G= I + Q + Q2 + Q2 + . . .

Note that Gij is the mean number of times that X = j given X0 = i. Properties Trouble with infinity
Fundamental Matrix calculations

4 Applications
4.1 Genetics
Definition 4.1. Here are some basic definitions. You don’t need to know very much detail though!
1. A gene is a small area on a chromosome;
2. A chromosome is an organised structure of DNA and protein found in cells;
3. A locus is the position of a gene in the chromosome;
4. An allele is one of several different forms of a particular gene;
5. Diploid reproduction is where an offspring takes one chromosome from each parent from a homol-
ogous pair;
ST202 Stochastic Processes 3

6. Genotype is the genetic composition of an individual;

7. Phenotype is the physical appearance of the genotype;
8. Autosomesare chromosomes other than sex chromosomes.
Theorem 4.2 (The Hardy-Weinberg Law).

Definition 4.3. The probability generating function of Z

4.2 Branching Process

4.3 Random Walks

5 Renewal Theory
5.1 Renewal Process
5.2 Delayed Renewal Process
5.3 Stationary Renewal Process
5.4 Coupling and the Renewal Theorem

6 Limiting Behaviour
Limits for n-step transition probabilities Stationary distributions

7 The Gibbs Sampler

The algorithm and a simple example The Ising Model