STA305 FST 17november2022 Topic2

Introduction
Chapman-Kolmogorov equation
Reducible Markov Chain
Stochastic Processes-Markov Chain
Ivivi Joseph Mwaniki

Department of Mathematics, University of Nairobi
Email: jimwaniki@uonbi.ac.ke
November 17, 2022
1/64
Ivivi Joseph Mwaniki The University of Nairobi DOM STA305 Stochastic ProcessesNovember 17, 2022
Introduction
Outline Reducible Markov Chain
1 Introduction
Stochastic process
Markov Process
Graphical representation
2 Chapman-Kolmogorov equation
Classification
Canonical form MC
Long term behaviour of MC
3 Reducible Markov Chain

MC classification
Exercises
R-code
2/64
Introduction Stochastic process Markov
Chapman-Kolmogorov equation Process
Reducible Markov Chain Graphical representation
Few Examples
3/64
Introduction Stochastic process
Markov Process
Markov Chain Graphical representation
Company Employment
0.6
Supervisor Mananger 0.9

0.2
0.8
0.1
4/64
Trainee
0.4
Markov Process
Markov Chain Graphical representation
Status of Weather
0.4
0.3
0.7
Sunny Rainy
0.6
5/64
Markov Process
Discrete state space Graphical representation
Poisson Process
A Poisson point process

N((s,t]) counts the number
of events occurring in the
interval (s,t]
A Poisson process X(t)
counts the number of events
occurring up to time t.
Formally X(t) = N((0,t])
6/64
Markov Process
Discrete state space Graphical representation
Poisson Process
A Poisson point process

N((s,t]) counts the number
of events occurring in the
interval (s,t]
A Poisson process X(t)
counts the number of events
occurring up to time t.
Formally X(t) = N((0,t])
It is a Markov process with
continuous time space and
discrete state space.
6/64
Markov Process
Continuous state space Graphical representation
Wiener process R Codes
# ivivi mwaniki 2020

# sta305 probability modeling
# yahoo finance historical data
#####################################
hdata<-read.csv(file.choose());
hdat<-hdata[,7]# adj.closing price
plot.ts(hdat,lwd=3,col=c(4),
main="Time plot RUT2000 index",
ylab="",xlab="days")
#####################################
7/64
Markov Process

#####################################
#####################################
7/64
Markov Process

#####################################
#####################################
7/64
Markov Process

#####################################
#####################################
7/64
Markov Process

#####################################
#####################################
7/64
Markov Process

#####################################
#####################################
7/64
Stochastic Process
8/64
Markov Process
Stochastic Processes Graphical representation
Definition (Stochastic Process)

A stochastic process is a family {X(t)}t∈T defined on the
same probability space (Ω, F, P) and taking their values in
some measurable space (E, E)
Let T be an arbitrary indexing set. It can be N (Natural

numbers),R (Real numbers) or R+ (the non-negative real
numbers)
9/64
Markov Process


numbers)
Therefore a stochastic process is a collection of random
variables indexed by time (in a wide sense) which may
record the evolution of some phenomenon.
9/64
Markov Process

numbers)
The set of possible values of a single random variable Xt
of a stochastic process {Xt}t∈T is known as its state
space.
9/64
Markov Process


numbers)
The set of possible values of a single random variable Xt
of a stochastic process {Xt}t∈T is known as its state
space.
The state space is discrete if it contains a finite or
denumerable infinity of points, otherwise it is continuous.
9/64
Markov Process
Multidimensional process Graphical representation
The stochastic process {X(t)}t∈T may be multidimensional.

10/64
Markov Process
The stochastic process {X(t)}t∈T may be
multidimensional. Consider {X(t)}t∈T = {(X1(t), X2(t))}t∈T
where X1(t)
represent daily maximum temperature and X2(t)
represent daily minimum temperature at a given place
observed over a period of time say (0,s)
10/64
Markov Process

where X1(t)
Then {X(t)}t∈T is an example of two dimensional
stochastic process
10/64
Markov Process

where X1(t)
Then {X(t)}t∈T is an example of two dimensional
stochastic process
One dimensional process can be classified further into
four categories.
10/64
Markov Process
Four kinds of stochastic processes Graphical representation
One dimensional stochastic process {Xt}t∈T can be classified

into four kinds, based on the state space and the indexing set
T.
Continuous time and continuous state space; Suppose
X(t) represents the maximum temperature at a particular
space in time (0,t), then the state space is continuous
and in a
continuous time.eg Wiener process or Brownin motion.
11/64
Markov Process
T.
and in a
Continuous time and discrete state space; i.e. possible
values of random variable Xt are discrete and the
indexing set
T ∈ R+ is continuous; e.g.Poisson process
11/64
Markov Process

T.
and in a
indexing set
Discrete time and continuous state space; i.e. possible
values of random variable Xt are continuous and the
indexing set T ∈ N
11/64
Markov Process

T.
and in a
indexing set
Discrete time and continuous state space; i.e. possible
values of random variable Xt are continuous and the
indexing set T ∈ N
Discrete time and discrete state space; i.e. possible
indexing set T ∈ N e.g.Discrete-time Markov chain.
11/64
Markov chain Ivivi Joseph Mwaniki The University of Nairobi DOM STA305
Stochastic ProcessesNovember 17, 2022
12/64
Markov Process
MC Road Map 2.1 Irreducible
Graphical representation Markov chains
1 Markov Chains
Stochastic Process
2.2 Reducible Markov chain
(Recurrent) states 3.3 Canonical form 3.5 Transient states
3.1 Transient states 3.6 Absorbing
3.4 Fundamental Matrix

4.1 Null 4.2 Non-null
3.2 Persistent
infinite Mean recurrence time Markov chain
4.4 Aperiodic 3.7 Reducible Chains

4.3 Periodic with more than one
non-null closed chain
finite mean recurrence
non-null
4.5 EGORDIC
Markov chain
Ivivi Joseph Mwaniki The University of Nairobi DOM STA305

13/64
Markov Process
Markov Chains Graphical representation
Definition (Markov Process)

A Markov process {Xt} is a stochastic process with the
property that, given the value of Xt, the values of Xs for s > t
are not influenced by the values of Xu for u < t.
Definition
A discrete-time Markov chain is a Markov process whose
state space is a finite or countable set, whose index set is
T = {0, 1, 2, 3, ...}.
In formal terms, the Markov property is that
Pr{Xn+1 = j|X0 = i0, ..., Xn−1 = in−1, Xn = i} = Pr{Xn+1 = j|Xn = i} for all
points n and states i0, ..., in−1, i, j
14/64
Markov Process
Definition (Markov Process)

A Markov process {Xt} is a stochastic process with the
property that, given the value of Xt, the values of Xs for s > t
are not influenced by the values of Xu for u < t.
Definition
A discrete-time Markov chain is a Markov process whose
state space is a finite or countable set, whose index set is
T = {0, 1, 2, 3, ...}.
In formal terms, the Markov property is that
Pr{Xn+1 = j|X0 = i0, ..., Xn−1 = in−1, Xn = i} = Pr{Xn+1 = j|Xn = i}
for all points n and states i0, ..., in−1, i, j

It is customary to speak of Xn being in state i if Xn = i.
14/64
Markov Process
To a pair of states (i, j) at the two successive trials there is

an associated conditional probability pij
15/64
Markov Process
The transition probability pij are basic to the study of
the structure of the Markov chain
15/64
Markov Process

The transition probability may or may not be independent
on n.
15/64
Markov Process

on n.
If the transition probability pij is independent of n, the
Markov chain is said to be homogeneous (to have
stationary probabilities)
15/64
Markov Process

on n.
If it is dependent on n, the chain is said to be
Non-homogeneous
15/64
Markov Process
on n.
If it is dependent on n, the chain is said to be
Non-homogeneous
m− step transition probability is defined and is denoted
by p(m)
jk = Pr{Xn+m = k|Xn = j}
15/64
Markov Process
The probability of Xn+1 being in state j given that Xn is in

state i is called the one step transition probability and is
denoted by
pn,n+1
ij = Pr{Xn+1 = j|Xn = i} = pij
16/64
Markov Process

denoted by
pn,n+1
Vast majority of Markov chains have stationary probabilities
16/64
Markov Process
denoted by
pn,n+1
Vast majority of Markov chains have stationary probabilities
It is customary to arrange these numbers pij in a matrix
form, in the infinite square array.
p00 p01 p02 . . p20 p21
.. p22 . . . . .
...
P=
p10 p11 p12
= ||pij ||
16/64
Markov Process
P is known as the transition probability matrix of the process.

17/64
Markov Process
P is known as the transition probability matrix of the
process. If the number of the states is finite the P is a finite
square matrix whose order is equal to the maximum
number of states.
17/64
Markov Process

number of states. The quantities pij satisfy the following
conditions
pij ≥ 0, for i, j = 0, 1, 2, ...

∞
X j=0
pij = 1 for i = 0, 1, 2, ...,

17/64
Markov Process

number of states. The quantities pij satisfy the following
conditions
pij ≥ 0, for i, j = 0, 1, 2, ...

∞
X j=0 pij = 1 for i = 0,
1, 2, ...,
A Markov process is completely

defined once its transition
probability matrix P and initial state
X0 are provided. 17/64

Markov Process
Order of Markov Chain Graphical representation
Order of Markov Chain
Definition
A Markov chain {Xn} is said to be of order s (s = 1, 2, 3, ...) if for all n
Pr{Xn = k|Xn−1 = j, Xn−2 = j1, ..., Xn−s = js−1, ...}

= Pr{Xn = k|Xn−1 = j, ..., Xn−s = js−1}
whenever left hand side is defined.
18/64
Markov Process
Order of Markov Chain Graphical representation
Order of Markov Chain

Definition
A Markov chain {Xn} is said to be of order s (s = 1, 2, 3, ...) if for all n
Pr{Xn = k|Xn−1 = j, Xn−2 = j1, ..., Xn−s = js−1, ...}

= Pr{Xn = k|Xn−1 = j, ..., Xn−s = js−1}
whenever left hand side is defined.
We confine our attention to Markov chain of order one.
Pr{Xn = k|Xn−1 = j, Xn−2 = j1, ..., X0 = jn−1}

= Pr{Xn = k|Xn−1 = j}
= pjk
18/64
Markov Process
Markov chain as graph Graphical representation
Consider a four state (A, B, C, D) Markov

chain {Xn}n∈N.
19/64
Markov Process
chain {Xn}n∈N.
We represent the following transition matrix
in a graphical format.
19/64
Markov Process

chain {Xn}n∈N.
We represent the following transition matrix
in a graphical format.
let p + q = 1, p ≥ 0, q ≥ 0
ABCD
2 2
A p (1 − p) p(1 − p) p(1 − p)
B0010
C0001
D q2 q(1 − q) q(1 − q) (1 − q)2
19/64
Markov Process
Markov Chain as graph Graphical representation
The four states A, B, C, D and their transition probability

can be represented as
20/64
Markov Process
The four states A, B, C, D and their transition probability
can be represented as
Patch H
2
(1 − p)
B
1
q(1 − q)
p(1 − p) (1 − q)2
2
pD
A C
q2 1
p(1 − p) q(1 − q)
20/64
Markov Process
Suppose that the chance of rain tomorrow

depends on previous weather conditions only
through whether or not it is raining today and
not on past weather conditions,
Markov chain graph

0.4
0.3
0.7
Sunny Rainy
0.6
21/64
Markov Process

If we say that the process is in state 0 when it
rains and state 1 when it does not rain, then the
chain whose transition probabilities are
Markov chain graph given by P
resulting process is a two-state Markov
0.4 0.6 0.7 0.3

0.4 P=
0.3
1 0.50 0.40 0.10
2 0.30 0.40 0.30
3 0.20 0.30 0.50
0.7
Sunny Rainy 0.6

Mood of an individual is considered as a three
state Markov chain having transition probability
matrix Q
123 21/64
Markov Process

If we say that the process is in state 0 when it
rains and state 1 when it does not rain, then the
chain whose transition probabilities are
Markov chain graph given by P
resulting process is a two-state Markov
0.4 0.6 0.7 0.3

0.4 P=
0.3
2 0.30 0.40 0.30
3 0.20 0.30 0.50
0.7 20
and the limiting distribution is Q
123
1 0.3387 0.3710 0.2903
Sunny Rainy 0.6 2 0.3387 0.3710 0.2903
3 0.3387 0.3710 0.2903
Mood of an individual is considered as a three
state Markov chain having transition probability
matrix Q
21/64
123
1 0.50 0.40 0.10
Markov Process
Markov chain graph

0.6
Supervisor Mananger 0.9

0.2
0.8
0.1
22/64
Trainee
0.4
Markov Process
Example1 Graphical representation
Example1: A Markov chain X0, X1, ... on states 0, 1, 2 has

the transition probability matrix
distribution
P= 0.1 0.4 0.1 0.5

0.2 0.7 0.1 0.8 0.1
and initial
p0 = Pr{X0 = 0} = 0.3, p1 = Pr{X0 = 1} = 0.4,

p2 = Pr{X0 = 2} = 0.3,
Determine
(i) Pr{X2 = 0|X1 = 2}, (ii)Pr{X0 = 0, X1 = 1, X2 = 2}, , (iii) Pr{X3 =

2, X1 = 2, X2 = 0|X0 = 1}
23/64
Markov Process
Example1. Graphical representation
Solution: A Markov chain X0, X1, ... on states 0, 1, 2 has the

transition probability matrix the on step transition is given by
Pr{X2 = 0|X1 = 2} = p20 = 0.1
24/64
Markov Process

Pr{X2 = 0|X1 = 2} = p20 = 0.1
Similarly, applying the Markov chain concept we get
Pr{X0 = 0, X1 = 1, X2 = 2} = Pr{X2 = 2, X1 = 1, X0 = 0}
= Pr{X2 = 2|X1 = 1}Pr{X1 = 1|X0 = 0}Pr{X0 = 0}
= p12p01p0 = p0p01p12
= 0.3 ∗ 0.2 ∗ .5 = .03
24/64
Markov Process

Pr{X2 = 0|X1 = 2} = p20 = 0.1
Similarly, applying the Markov chain concept we get
Pr{X0 = 0, X1 = 1, X2 = 2} = Pr{X2 = 2, X1 = 1, X0 = 0}
= Pr{X2 = 2|X1 = 1}Pr{X1 = 1|X0 = 0}Pr{X0 = 0}
= p12p01p0 = p0p01p12
= 0.3 ∗ 0.2 ∗ .5 = .03
event {X2 = 0, X3 = 2, X1 = 2|X0 = 1} can be rearranged

Pr{X3 = 2, X2 = 0, X1 = 2|X0 = 1}
= Pr{X3 = 2|X2 = 0}Pr{X2 = 0|X1 = 2}Pr{X1 = 2|X0 = 1}
= p02p20p12 = p12p20p02 = 0.5 ∗ 0.1 ∗ 0.7
= 0.035
24/64
Markov Process
Exercise Graphical representation
Let {Xm}m≥0 be a Markov chain with three states 0, 1, 2. and

with transition matrix
0
P= 3/4 1/4
1/4 0
3/4 1/4
1/4 1/2
and initial distribution and the initial

distribution
Pr{X0 = i} = 1/3, i = 0, 1, 2
Determine
(i) Pr{X2 = 1|X1 = 0}, (ii) Pr{X2 = 1,

X1 = 2|X0 = 2}, (iii) Pr{X3 = 1, X2 =
2, X1 = 1, X0 = 2} 25/64
(iv) Pr{X1 = 0, X2 = 1, X0 = 2}

Introduction Classification
Canonical form MC
Higher transition probabilities Long term behaviour of MC
P(n)
ij denotes the probability that the process goes from state
i to state j in n transitions.
P(n) = ||p(n)
ij ||
(n)
p
ij = Pr{Xm+n = j|Xm = i}
26/64
Canonical form MC
P(n)
P(n) = ||p(n)
ij ||
(n)
p
The n−step transition probabilities of a Markov chain
satisfy ij =X∞
p(n)
k=0
pik p(n−1) kj .
26/64
Canonical form MC
P(n)
P(n) = ||p(n)
ij ||
(n)
p
satisfy ij =X∞
p(n) pik p(n−1)

k=0
kj .
From the theory of matrices we recognize that P(n) =

PP(n−1),i.e. the n-step transition probabilities p(n)
the entries in the matrix Pn, the
nth power of P 26/64
ij are
Canonical form MC
n-step transition Long term behaviour of MC
We prove the following theorem

27/64
Canonical form MC
satisfy ij =X∞
p(n)
k=0
pik p(n−1) kj .
27/64
Canonical form MC

satisfy ij =X∞
pik p(n−1) p(n)

kj . p(n)
k=0
It is well known that
ij = Pr{Xn = j|X0 = i}
∞ ∞
=X =X
k=0 k=0
∞
=X k=0
Pr{Xn = j, x1 = k|X0 = i}
Pr{Xn = j|X0 = i}Pr{Xn = j|x0 =

i, X1 = k}
27/64
pik p(n−1)
kj
Canonical form MC
Chapman-Kolmogorov equationLong term behaviour of MC
In general, we have
p(m+n)
ij =X ir =X
p(n)
(m)
rj p
r
(n)
p
(m)
ir p
rj
r
28/64
Canonical form MC
In general, we have
p(m+n)
ij =X ir =X
p(n)
(m) (m)
rj p ir p
r rj
p(n) r
The equation is a special case of

which is satisfied by the transition
probabilities of a Markov chain.

28/64
Canonical form MC
In general, we have
p(m+n)
ij =X ir =X
p(n)
(m) (m)
rj p ir p
r rj
p(n) r
The equation is a special case of

which is satisfied by the transition
probabilities of a Markov chain.
From the general equation above,
rj , for any r.
we get
p(m+n)
(m)
ij ≥ p
(n)
ir p
28/64
Canonical form MC
In general, we have
p(m+n)
ij =X ir =X
p(n)
(m) (m)
rj p ir p
r rj
p(n) r
The equation is a special case of Chapman-Kolmogorov

equation which is satisfied by the transition probabilities
of a Markov chain.
From the general equation above, we get
p(m+n)
(m)
ij ≥ p
(n)
ir p
rj , for any r.
If the probability process initially being in state j is pj =
Pr{X0 = j}, then the probability of the process being in
state k at time n is
∞
k =X
(n)
p
j=0
pjp(n) 28/64
jk = Pr{Xn = k}
Canonical form MC
MC Examples1 Long term behaviour of MC
We consider Graduate trainee,supervisor and

manager a Markov Chain example with transition
probability matrix
0.6 0 0.16 0.62 0.62
0 .8 .2   2
P= P = 0.12 0.02 
0.1 0 0.72 0.06
 
0.9 0.4 0.36 0.32
0.396 0.396
0.258 0.447 
0.428 0.214
P5 = 0.314
 0.390
0.157


P25 =

0.2 0.4 0.4
0.2 0.4 0.4
0.2 0.4 0.4
29/64
Canonical form MC
We consider Graduate trainee,supervisor and

manager a Markov Chain example with transition
probability matrix
0.6 0 0.16 0.62 0.62
0 .8 .2  
P= P2 = 0.12 0.02 
0.1 0 0.72 0.06
 
0.9 0.4 0.36 0.32
0.428 0.396 0.2 0.4
0.314   0.4 0.2
P5 = 0.157 0.4 0.4
 0.396 0.2 0.4
0.447 25
0.4
0.214 P =  
0.258
0.390  
We note that Pr{X2 = 0|X0 = 0} =

22 = 0.4, etc
p(2)
00 = 0.16 Similarly Ivivi Joseph Mwaniki The University of Nairobi DOM STA305
(5) Stochastic ProcessesNovember 17, 2022
p
12 = Pr{X6 = 2|X1 = 1} = 0.447; p(25)
29/64
Canonical form MC
We consider a Markov Chain with transition probability matrix

0.7 0.3 0.40 0.82 0.51
010   2
P = 0.00 0.18 
P= 0.4 0 0.60 0.28

 0.6 0 0.00 0.21
 0.16905
0.17722
0.46023
0.44304
20 100   0.37072
P =P = 0.37975
0.18222
0.18691 0.17722
0.43250
0.42263 0.44304
0.38528
0.39047 0.37975
0.17722
0.44304
0.37975




30/64
Canonical form MC
We consider a Markov Chain with transition probability matrix

0.7 0.3 0.40 0.82 0.51
010  2
 P = 0.00 0.18 
P= 0.4 0 0.60 0.28

 0.6 0 0.00 0.21
 0.37072 0.37975
0.18222 0.17722
 0.43250 0.44304
P20 =
0.38528 0.37975
0.18691 
0.17722
0.42263 0.44304
P100 = 0.39047 0.37975  
0.16905 0.17722
0.46023 0.44304
p(5)
We note that Pr{X2 = 0|X0 = 0} = (100)
02 = Pr{X5 = 2|X0 = 0} = 0.81; p
p(2)
00 = 0.16. Similarly
00 = 0.17721, etc
30/64
Canonical form MC
R-code Long term behaviour of MC
############################
#matrix power Markov Chains
############################
matrixpower <- function(mat,k){
out <- mat
for (i in 2:k) {
out <- out %*% mat
}
out
}
mat<-matrix(c(0.1,0.9,0.6,0.4),nrow=2,byrow=T);mat
mat1<-matrix(c(0.0,1.0,0.0,0.4,0,0.6,0.0,0.7,0.3),nrow=3,byrow=T);mat1
mat2<-matrix(c(0.0,0.5,0.5,0.25,0.75,0,0.5,0.5,0),nrow=3,byrow=T);mat2
mat3<-matrix(c(0.5,0.5,0,0,0.25,0.75,0,0,0,0,1/3,2/3,0,0,2/3,1/3),nrow=4,byrow=T);mat3
mat4<-matrix(c(0,1,0,0,0,0,1,0,0,0,0,1,0.5,0,0.5,0),nrow=4,byrow=T);mat4
mat5<-matrix(c(1/3,2/3,0,0,1,0,0,0,1/2,0,1/2,0,0,0,1/2,1/2),nrow=4,byrow=T);mat5
matrixpower(mat,2)
matrixpower(mat4,28)
############################
m<-200
set1<-array(NA,m)
set1[1]<-1.0
matrixpower(mat4,2)
for(j in 2:m){
pr<-matrixpower(mat4,j)
set1[j]<-pr[1,2]
print(pr)
#return(set1)
}
31/64
Canonical form MC
Exercises Long term behaviour of MC
A Markov Chain {Xn} on the states 0, 1, 2. has the

following transition matrix, say P. In each of the following
transition matrices, compute the two step transition
matrix,P2,and
P4, P16, P50
32/64
Canonical form MC

matrix,P2,and
P4, P16, P50
Determine the conditional probabilities
Pr{X3 = 1|X0 = 0}, Pr{X30 = 2|X1 = 2} and
Pr{X48 = 1|X8 = 0}; Pr{Xn+15 = 1|Xn−1 = 2} for
n = 2, 10, 38, 73, 99.
32/64
Canonical form MC

matrix,P2,and P4, P16, P50
Determine the conditional probabilities
Pr{X3 = 1|X0 = 0}, Pr{X30 = 2|X1 = 2} and
Pr{X48 = 1|X8 = 0}; Pr{Xn+15 = 1|Xn−1 = 2} for
n = 2, 10, 38, 73, 99.
The corresponding transition probabilities P are as follows
0.5 0.2 0.3 0.5 0.3 0.2
3/4 1/4 ; 0.3 0.6 0.1
0 0.7 0.2 0.1 ; 0.5 0 0.5
1/4 1/2 1/4 0 0 0.6 0.4 0.1 0.2 0.7
0.5 0 0.5 0.6 0.2 0.2
3/4 1/4
0.3 0.6 0.1
0 1/2 1/2 ;
1/2 0 1/2 1/2 0.6 0.2 0.2 ;
1/2 0 0.3 0.3 0.4
32/64
MC
Canonical form MC
2.1 Irreducible
classification Long term behaviour of Markov chains

MC
2.2 Reducible Markov chain

1 Markov Chains
Stochastic Process
(Recurrent) states 3.3 Canonical form 3.5 Transient states
3.1 Transient states 3.6 Absorbing
3.4 Fundamental Matrix

4.1 Null 4.2 Non-null
3.2 Persistent
infinite Mean recurrence time Markov chain
4.4 Aperiodic 3.7 Reducible Chains

4.3 Periodic with more than one
non-null closed chain
finite mean recurrence
non-null
4.5 EGORDIC
Markov chain
33/64

Canonical form MC
Markov Chain classification Long term behaviour of MC
A communicating class of states is closed if it is not

possible to leave that class.
34/64
Canonical form MC
Markov Chain classification Long term behaviour of MC
A communicating class of states is closed if it is not
possible to leave that class.
A state is said to be an absorbing if the set {i} is a
closed class.
34/64

STA305 FST 17november2022 Topic2

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

STA305 FST 17november2022 Topic2

Uploaded by

Copyright:

Available Formats

Introduction

Stochastic Processes-Markov Chain

Ivivi Joseph Mwaniki

November 17, 2022

3 Reducible Markov Chain

Supervisor Mananger 0.9

A Poisson point process

A Poisson point process

Wiener process R Codes

# ivivi mwaniki 2020

Wiener process R Codes

# ivivi mwaniki 2020

Wiener process R Codes

# ivivi mwaniki 2020

Wiener process R Codes

# ivivi mwaniki 2020

Wiener process R Codes

# ivivi mwaniki 2020

Definition (Stochastic Process)

Let T be an arbitrary indexing set. It can be N (Natural

Definition (Stochastic Process)

Let T be an arbitrary indexing set. It can be N (Natural

Definition (Stochastic Process)

Definition (Stochastic Process)

Let T be an arbitrary indexing set. It can be N (Natural

The stochastic process {X(t)}t∈T may be multidimensional.

The stochastic process {X(t)}t∈T may be

The stochastic process {X(t)}t∈T may be

One dimensional stochastic process {Xt}t∈T can be classified

One dimensional stochastic process {Xt}t∈T can be classified

One dimensional stochastic process {Xt}t∈T can be classified

3.1 Transient states 3.6 Absorbing

3.4 Fundamental Matrix

4.4 Aperiodic 3.7 Reducible Chains

Ivivi Joseph Mwaniki The University of Nairobi DOM STA305

Definition (Markov Process)

points n and states i0, ..., in−1, i, j

Definition (Markov Process)

In formal terms, the Markov property is that

Pr{Xn+1 = j|X0 = i0, ..., Xn−1 = in−1, Xn = i} = Pr{Xn+1 = j|Xn = i}

for all points n and states i0, ..., in−1, i, j

To a pair of states (i, j) at the two successive trials there is

To a pair of states (i, j) at the two successive trials there is

To a pair of states (i, j) at the two successive trials there is

To a pair of states (i, j) at the two successive trials there is

The probability of Xn+1 being in state j given that Xn is in

The probability of Xn+1 being in state j given that Xn is in

P is known as the transition probability matrix of the process.

P is known as the transition probability matrix of the

pij ≥ 0, for i, j = 0, 1, 2, ...

pij = 1 for i = 0, 1, 2, ...,

P is known as the transition probability matrix of the

pij ≥ 0, for i, j = 0, 1, 2, ...

A Markov process is completely

Ivivi Joseph Mwaniki The University of Nairobi DOM STA305

Pr{Xn = k|Xn−1 = j, Xn−2 = j1, ..., Xn−s = js−1, ...}

whenever left hand side is defined.

Order of Markov Chain

Pr{Xn = k|Xn−1 = j, Xn−2 = j1, ..., Xn−s = js−1, ...}

whenever left hand side is defined.

We confine our attention to Markov chain of order one.

Pr{Xn = k|Xn−1 = j, Xn−2 = j1, ..., X0 = jn−1}

Consider a four state (A, B, C, D) Markov