You are on page 1of 185

Discrete Time Markov Chains, Definition

and classification
Bo Friis Nielsen1
1 Applied

Bo Friis Nielsen

Today:
I
I
I
I

Short recap of probability theory

Markov chain introduction (Markov property)
Chapmann-Kolmogorov equations
First step analysis

Next week
I
I
I
I

Random walks
First step analysis revisited
Branching processes
Generating functions

Two weeks from now

I
I
I

Classification of states
Classification of chains
Discrete time Markov chains - invariant probability
distribution
Bo Friis Nielsen

Bo Friis Nielsen

Sample space

Bo Friis Nielsen

Sample space

Bo Friis Nielsen

Sample space
Outcome

Bo Friis Nielsen

Sample space
Outcome

Bo Friis Nielsen

Sample space
Outcome
Event

Bo Friis Nielsen

Sample space
Outcome
Event

A, B

Bo Friis Nielsen

Basic concepts in probability

Sample space
Outcome
Event
Complementary event

A, B

Bo Friis Nielsen

Basic concepts in probability

Sample space
Outcome
Event
Complementary event

A, B
Ac =

Bo Friis Nielsen

Basic concepts in probability

Sample space
Outcome
Event
Complementary event

A, B
Ac = \A

Bo Friis Nielsen

Basic concepts in probability

Sample space
Outcome
Event
Complementary event

A, B
Ac = \A

Union

Bo Friis Nielsen

Basic concepts in probability

Sample space
Outcome
Event
Complementary event
Union

A, B
Ac = \A

AB

Bo Friis Nielsen

Basic concepts in probability

Sample space
Outcome
Event
Complementary event
Union

A, B
Ac = \A

AB

A or B

Bo Friis Nielsen

Basic concepts in probability

Sample space
Outcome
Event
Complementary event
Union

A, B
Ac = \A

AB

A or B

Intersection

Bo Friis Nielsen

Basic concepts in probability

Sample space
Outcome
Event
Complementary event

A, B
Ac = \A

Union

AB

A or B

Intersection

AB

Bo Friis Nielsen

Basic concepts in probability

Sample space
Outcome
Event
Complementary event

A, B
Ac = \A

Union

AB

Intersection

AB

A or B
Outcome is in
both A and B

Bo Friis Nielsen

Basic concepts in probability

Sample space
Outcome
Event
Complementary event

A, B
Ac = \A

Union

AB

Intersection

AB

A or B
Outcome is in
both A and B

Bo Friis Nielsen

Basic concepts in probability

Sample space
Outcome
Event
Complementary event

A, B
Ac = \A

Union

AB

Intersection

AB

A or B
Outcome is in
both A and B

Bo Friis Nielsen

0 P(A) 1,

Bo Friis Nielsen

P() = 1

0 P(A) 1,

P() = 1

Bo Friis Nielsen

for A B =

0 P(A) 1,

P() = 1

for A B =

Bo Friis Nielsen

0 P(A) 1,

P() = 1

for A B =

P() = 0,

P(Ac ) = 1 P(A)

Bo Friis Nielsen

0 P(A) 1,

P() = 1

for A B =

P() = 0,

P(Ac ) = 1 P(A)

P(A B) = P(A) + P(B) P(A B)

(inclusion- exlusion)

Bo Friis Nielsen

P(A|B) =

P(A B)
P(B)

Bo Friis Nielsen

Conditional probability and independence

P(A|B) =

P(A B)
P(A B) = P(A|B)P(B)
P(B)

Bo Friis Nielsen

Conditional probability and independence

P(A|B) =

P(A B)
P(A B) = P(A|B)P(B)
P(B)

(multiplication rule)
i Bi =

Bo Friis Nielsen

Conditional probability and independence

P(A|B) =

P(A B)
P(A B) = P(A|B)P(B)
P(B)

(multiplication rule)
i Bi =

Bi Bj =

Bo Friis Nielsen

i 6= j

Conditional probability and independence

P(A|B) =

P(A B)
P(A B) = P(A|B)P(B)
P(B)

(multiplication rule)
i Bi =
P(A) =

Bi Bj =

P(A|Bi )P(Bi )

i 6= j

(law of total probability)

Independence:
P(A|B) = P(A|B c ) = P(A)

Bo Friis Nielsen

Conditional probability and independence

P(A|B) =

P(A B)
P(A B) = P(A|B)P(B)
P(B)

(multiplication rule)
i Bi =
P(A) =

Bi Bj =

P(A|Bi )P(Bi )

i 6= j

(law of total probability)

Independence:
P(A|B) = P(A|B c ) = P(A) P(A B) = P(A)P(B)

Bo Friis Nielsen

Discrete random variables

Mapping from sample space to metric space
Probability mass function
f (x) = P(X = x) = P ({|X () = x})
Distribution function
F (x) = P(X x) = P ({|X () x}) =

f (t)

tx

Expectation
X
X
X
xP(X = x), E(g(X )) =
g(x)P(X = x) =
g(x)f (x)
E(X ) =
x

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Joint distribution

f (x1 , x2 ) = P(X1 = x1 , X2 = x2 ),
F (x1 , x2 ) = P(X1 x1 , X2 x2 )
X
X
fX1 (x1 ) = P(X1 = x1 ) =
P(X1 = x1 , X2 = x2 ) =
f (x1 , x2 )
x2

FX1 (x1 ) =

x2

P(X1 = t1 , X2 = x2 ) = F (x1 , )

tx1 ,x2

Bo Friis Nielsen

We can define the joint distribution of (X0 , X1 ) through

P(X0 = x0 , X1 = x1 ) = P(X0 = x0 )P(X1 = x1 |X0 = x0 ) = P(X0 = x0 )Px0 ,x1
Suppose now some stationarity in addition that X2 conditioned
on X1 is independent on X0
P(X0 = x0 , X1 = x1 , X2 = x2 ) =
P(X0 = x0 )P(X1 = x1 |X0 = x0 )P(X2 = x2 |X0 = x0 , X1 = x1 ) =
P(X0 = x0 )P(X1 = x1 |X0 = x0 )P(X2 = x2 |X1 = x1 ) =
px0 Px0 ,x1 Px1 ,x2
which generalizes to arbitrary n.

Bo Friis Nielsen

Markov property
P (Xn = xn |H)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Markov property
P (Xn = xn |H)
= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Markov property
P (Xn = xn |H)
= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )
= P (Xn = xn |Xn1 = xn1 )

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Markov property
P (Xn = xn |H)
= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )
I

= P (Xn = xn |Xn1 = xn1 )

Generally the next state depends on the current state and
the time

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Markov property
P (Xn = xn |H)
= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )
I

= P (Xn = xn |Xn1 = xn1 )

Generally the next state depends on the current state and
the time
In most applications the chain is assumed to be time
homogeneous, i.e. it does not depend on time

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Markov property
P (Xn = xn |H)
= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )
I

= P (Xn = xn |Xn1 = xn1 )

Generally the next state depends on the current state and
the time

In most applications the chain is assumed to be time

homogeneous, i.e. it does not depend on time

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Markov property
P (Xn = xn |H)
= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )
I

= P (Xn = xn |Xn1 = xn1 )

Generally the next state depends on the current state and
the time

In most applications the chain is assumed to be time

homogeneous, i.e. it does not depend on time

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Markov property
P (Xn = xn |H)
= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )
I

= P (Xn = xn |Xn1 = xn1 )

Generally the next state depends on the current state and
the time

In most applications the chain is assumed to be time

homogeneous, i.e. it does not depend on time

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Markov property
P (Xn = xn |H)
= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )
I

= P (Xn = xn |Xn1 = xn1 )

Generally the next state depends on the current state and
the time

In most applications the chain is assumed to be time

homogeneous, i.e. it does not depend on time

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Markov property
P (Xn = xn |H)
= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )
I

= P (Xn = xn |Xn1 = xn1 )

Generally the next state depends on the current state and
the time

In most applications the chain is assumed to be time

homogeneous, i.e. it does not depend on time

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Markov property
P (Xn = xn |H)
= P (Xn = xn |X0 = x0 , X1 = x1 , X2 = x2 , . . . Xn1 = xn1 )
I

= P (Xn = xn |Xn1 = xn1 )

Generally the next state depends on the current state and
the time

In most applications the chain is assumed to be time

homogeneous, i.e. it does not depend on time

Bo Friis Nielsen

Bo Friis Nielsen

A profuse number of applications

Storage/inventory models

Bo Friis Nielsen

A profuse number of applications

Storage/inventory models

Telecommunications systems

Bo Friis Nielsen

A profuse number of applications

Storage/inventory models

Telecommunications systems

Biological models

Bo Friis Nielsen

A profuse number of applications

Storage/inventory models

Telecommunications systems

Biological models

Xn could be

Bo Friis Nielsen

A profuse number of applications

Storage/inventory models

Telecommunications systems

Biological models

Xn could be

Bo Friis Nielsen

A profuse number of applications

Storage/inventory models

Telecommunications systems

Biological models

Xn could be

I
I

The number of cars in stock

The number of days since last rainfall

Bo Friis Nielsen

A profuse number of applications

Storage/inventory models

Telecommunications systems

Biological models

Xn could be

I
I
I

The number of cars in stock

The number of days since last rainfall
The number of passengers booked for a flight

Bo Friis Nielsen

A profuse number of applications

Storage/inventory models

Telecommunications systems

Biological models

Xn could be

I
I
I
I

The number of cars in stock

The number of days since last rainfall
The number of passengers booked for a flight
See textbook for further examples

Bo Friis Nielsen

Example 1: Random walk with two

reflecting barriers 0 and N

Bo Friis Nielsen

Example 1: Random walk with two

reflecting barriers 0 and N

P=

Bo Friis Nielsen

Example 1: Random walk with two

reflecting barriers 0 and N

1 p

P =

Bo Friis Nielsen

Example 1: Random walk with two

reflecting barriers 0 and N

1 p

P =

Bo Friis Nielsen

Example 1: Random walk with two

reflecting barriers 0 and N

1 p

P =

Bo Friis Nielsen

...

Example 1: Random walk with two

reflecting barriers 0 and N

1 p

q

P =

Bo Friis Nielsen

...

Example 1: Random walk with two

reflecting barriers 0 and N

1 p
p

q
1

p
q

P =

Bo Friis Nielsen

...

Example 1: Random walk with two

reflecting barriers 0 and N

1 p
p

q
1

p
q

P =

0
p

Bo Friis Nielsen

...

Example 1: Random walk with two

reflecting barriers 0 and N

1 p
p

q
1

p
q

P =

0
p

Bo Friis Nielsen

...
...

0
0

0
0

0
0

Example 1: Random walk with two

reflecting barriers 0 and N

1 p
p
0

q
1

q
p

0
q
1pq

P = .
..
..
.
.
.
.

Bo Friis Nielsen

...
...
...
..
.

0
0
0
..
.

0
0
0
..
.

0
0
0
..
.

Example 1: Random walk with two

reflecting barriers 0 and N

1 p
p
0

q
1

q
p

0
q
1pq

P = .
..
..
.
.
.
.

0
0
0

Bo Friis Nielsen

...
...
...
..
.

0
0
0
..
.

0
0
0
..
.

0
0
0
..
.

...

Example 1: Random walk with two

reflecting barriers 0 and N

1 p
p
0

q
1

q
p

0
q
1pq

P = .
..
..
.
.
.
.

0
0
0

Bo Friis Nielsen

...
...
...
..
.

0
0
0
..
.

0
0
0
..
.

...

q 1pq

0
0
0
..
.
p

Example 1: Random walk with two

reflecting barriers 0 and N

1 p
p
0

q
1

q
p

0
q
1pq

P = .
..
..
.
.
.
.

0
0
0

0
0
0

Bo Friis Nielsen

...
...
...
..
.

0
0
0
..
.

0
0
0
..
.

...
...

q 1pq
0

0
0
0
..
.
p

Example 1: Random walk with two

reflecting barriers 0 and N

1 p
p
0

q
1

q
p

0
q
1pq

P = .
..
..
.
.
.
.

0
0
0

0
0
0

Bo Friis Nielsen

...
...
...
..
.

0
0
0
..
.

0
0
0
..
.

...
...

q 1pq
0
q

0
0
0
..
.
p

Example 1: Random walk with two

reflecting barriers 0 and N

1 p
p
0

q
1

q
p

0
q
1pq

P = .
..
..
.
.
.
.

0
0
0

0
0
0

Bo Friis Nielsen

...
...
...
..
.
...
...

q 1pq
p
0
q
1 q
0
0
0
..
.

0
0
0
..
.

0
0
0
..
.

Example 2: Random walk with one

reflecting barrier at 0

Bo Friis Nielsen

Example 2: Random walk with one

reflecting barrier at 0

P=

Bo Friis Nielsen

Example 2: Random walk with one

reflecting barrier at 0

1 p

P =

Bo Friis Nielsen

0 ...

Example 2: Random walk with one

reflecting barrier at 0

1 p
p

q
1

p
q

P =

Bo Friis Nielsen

0
p

0
0

0 ...
0 ...

Example 2: Random walk with one

reflecting barrier at 0

1 p
p
0
0

q
1

q
p
0

0
q
1pq
p
P =
0
0
q
1

p
q

Bo Friis Nielsen

0
0
0
p

...
...
...
...

Example 2: Random walk with one

reflecting barrier at 0

1 p
p
0
0

q
1

q
p
0

0
q
1pq
p
P =
0
0
q
1

p
q

.
.
.
.
..
..
..
..

Bo Friis Nielsen

0
0
0
p
..
.

...
...
...
...
...

Example 3: Random walk with two

absorbing barriers

P=

Bo Friis Nielsen

Example 3: Random walk with two

absorbing barriers

1

P =

Bo Friis Nielsen

Example 3: Random walk with two

absorbing barriers

1

P =

Bo Friis Nielsen

0 ...

Example 3: Random walk with two

absorbing barriers

1
0

q 1 p q

P =

0
p

Bo Friis Nielsen

0
0

0 ...
0 ...

0
0

0
0

0
0

Example 3: Random walk with two

absorbing barriers

P =

0
0
0
1
q 1pq
p
0
0
q
1pq
p
0
0
q
1pq
..
..
..
..
.
.
.
.

Bo Friis Nielsen

0
0
0
p
..
.

...
...
...
...
..
.

0
0
0
0
..
.

0
0
0
0
..
.

0
0
0
0
..
.

Example 3: Random walk with two

absorbing barriers

P =

0
0
0
1
q 1pq
p
0
0
q
1pq
p
0
0
q
1pq
..
..
..
..
.
.
.
.

0
0
0
p
..
.

0 ...

Bo Friis Nielsen

...
...
...
...
..
.

0
0
0
0
..
.

0
0
0
0
..
.

0
0
0
0
..
.

q 1pq p

Example 3: Random walk with two

absorbing barriers

P =

0
0
0
1
q 1pq
p
0
0
q
1pq
p
0
0
q
1pq
..
..
..
..
.
.
.
.

0
0
0
p
..
.

0
0

0 ...
0 ...

0
0

0
0

Bo Friis Nielsen

0
0

...
...
...
...
..
.

q 1 p q p
0
0
1
0
0
0
0
..
.

0
0
0
0
..
.

0
0
0
0
..
.

Bo Friis Nielsen

Bo Friis Nielsen

The matrix can be finite (if the Markov chain is finite)

p1,1 p1,2 p1,3 . . . p1,n

p2,1 p2,2 p2,3 . . . p2,n

P = p3,1 p3,2 p3,3 . . . p3,n
..
..
..
..
..
.
.
.
.
.

pn,1 pn,2 pn,3 . . . pn,n

Bo Friis Nielsen

The matrix can be finite (if the Markov chain is finite)

p1,1 p1,2 p1,3 . . . p1,n

p2,1 p2,2 p2,3 . . . p2,n

P = p3,1 p3,2 p3,3 . . . p3,n
..
..
..
..
..
.
.
.
.
.

pn,1 pn,2 pn,3 . . . pn,n
Two reflecting/absorbing barriers

Bo Friis Nielsen

or infinite

Bo Friis Nielsen

Bo Friis Nielsen

or infinite (if the Markov chain is infinite)

p1,1 p1,2 p1,3 . . .

p2,1 p2,2 p2,3 . . .

p3,1 p3,2 p3,3 . . .

..
..
..
P = ..
.
.
.
.
pn,1 pn,2 pn,3 . . .

.
..
..
..
..
.
.
.

Bo Friis Nielsen

p1,n . . .
p2,n . . .
p3,n . . .
..
..
.
.
pn,n . . .
..
..
.
.

or infinite (if the Markov chain is infinite)

p1,1 p1,2 p1,3 . . .

p2,1 p2,2 p2,3 . . .

p3,1 p3,2 p3,3 . . .

..
..
..
P = ..
.
.
.
.
pn,1 pn,2 pn,3 . . .

.
..
..
..
..
.
.
.

p1,n . . .
p2,n . . .
p3,n . . .
..
..
.
.
pn,n . . .
..
..
.
.

Bo Friis Nielsen

Bo Friis Nielsen

Bo Friis Nielsen

Bo Friis Nielsen

the statistical descriptor of the quantitative behaviour

a collection of discrete probability distributions

Bo Friis Nielsen

the statistical descriptor of the quantitative behaviour

a collection of discrete probability distributions

Bo Friis Nielsen

the statistical descriptor of the quantitative behaviour

a collection of discrete probability distributions

I
I

For each i we have a conditional distribution

What is the probability of the next state being j knowing that
the current state is i

Bo Friis Nielsen

the statistical descriptor of the quantitative behaviour

a collection of discrete probability distributions

I
I

For each i we have a conditional distribution

What is the probability of the next state being j knowing that
the current state is i pij = P (Xn = j|Xn1 = i)

Bo Friis Nielsen

the statistical descriptor of the quantitative behaviour

a collection of discrete probability distributions

I
I

For each i we have a conditional distribution

What is the probability of the next state being j knowing that
the
P current state is i pij = P (Xn = j|Xn1 = i)
j pij = 1

Bo Friis Nielsen

the statistical descriptor of the quantitative behaviour

a collection of discrete probability distributions

I
I

I
I

For each i we have a conditional distribution

What is the probability of the next state being j knowing that
the
P current state is i pij = P (Xn = j|Xn1 = i)
j pij = 1
We say that P is a stochastic matrix

Bo Friis Nielsen

Bo Friis Nielsen

and onwards

Bo Friis Nielsen

We have defined rules for the behaviour from one value

and onwards
Boundary conditions

Bo Friis Nielsen

We have defined rules for the behaviour from one value

and onwards
Boundary conditions specify e.g. behaviour of X0

Bo Friis Nielsen

We have defined rules for the behaviour from one value

and onwards
Boundary conditions specify e.g. behaviour of X0
I

X0 could be certain

Bo Friis Nielsen

We have defined rules for the behaviour from one value

and onwards
Boundary conditions specify e.g. behaviour of X0
I

X0 could be certain X0 = a

Bo Friis Nielsen

We have defined rules for the behaviour from one value

and onwards
Boundary conditions specify e.g. behaviour of X0
I
I

X0 could be certain X0 = a
or random

Bo Friis Nielsen

We have defined rules for the behaviour from one value

and onwards
Boundary conditions specify e.g. behaviour of X0
I
I

X0 could be certain X0 = a
or random P (X0 = i) = pi

Bo Friis Nielsen

We have defined rules for the behaviour from one value

and onwards
Boundary conditions specify e.g. behaviour of X0
I
I
I

X0 could be certain X0 = a
or random P (X0 = i) = pi
Once again we collect the possibly infinite many
parameters in a vector p

Bo Friis Nielsen

Bo Friis Nielsen

n step transition probabilities

P (Xn = j|X0 = i)

Bo Friis Nielsen

(n)

Bo Friis Nielsen

(n)

I

started in i

Bo Friis Nielsen

(n)

I

started in i

Bo Friis Nielsen

(n)

I

started in i

(n)

Bo Friis Nielsen

(n)

I

started in i

(n)

Bo Friis Nielsen

(n)

I

started in i

(n)

Bo Friis Nielsen

(n)

I

started in i

(P n )

(n)

Bo Friis Nielsen

Small example

1 p

q
P =
0
0

p
0
q
0

0
0
p
0
0
p
q 1q

Bo Friis Nielsen

Small example

1 p

q
P =
0
0

p
0
q
0

0
0
p
0
0
p
q 1q

P (2) =

Bo Friis Nielsen

Small example

1 p

q
P =
0
0

P (2)

p
0
q
0

0
0
p
0
0
p
q 1q

(1 p)2

=

Bo Friis Nielsen

Small example

1 p

q
P =
0
0

P (2)

p
0
q
0

0
0
p
0
0
p
q 1q

(1 p)2 + pq

=

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Small example

1 p

q
P =
0
0

P (2)

p
0
q
0

0
0
p
0
0
p
q 1q

(1 p)2 + pq (1 p)p

=

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

Small example

1 p

q
P =
0
0

P (2)

p
0
q
0

0
0
p
0
0
p
q 1q

(1 p)2 + pq (1 p)p

=

Bo Friis Nielsen

p2

Discrete Time Markov Chains, Definition and classification

Small example

1 p

q
P =
0
0

(2)

p
0
q
0

0
0
p
0
0
p
q 1q

(1 p)2 + pq (1 p)p

=

Bo Friis Nielsen

p2

Discrete Time Markov Chains, Definition and classification

Small example

1 p

q
P =
0
0

(2)

p
0
q
0

0
0
p
0
0
p
q 1q

(1 p)2 + pq (1 p)p

q(1 p)
=

Bo Friis Nielsen

p2

Discrete Time Markov Chains, Definition and classification

Small example

1 p

q
P =
0
0

(2)

p
0
q
0

0
0
p
0
0
p
q 1q

(1 p)2 + pq (1 p)p

q(1 p)
2qp
=

Bo Friis Nielsen

p2

Discrete Time Markov Chains, Definition and classification

Small example

1 p

q
P =
0
0

(2)

p
0
q
0

0
0
p
0
0
p
q 1q

(1 p)2 + pq (1 p)p

q(1 p)
2qp
=

Bo Friis Nielsen

p2
0

Discrete Time Markov Chains, Definition and classification

Small example

1 p

q
P =
0
0

(2)

p
0
q
0

0
0
p
0
0
p
q 1q

(1 p)2 + pq (1 p)p

q(1 p)
2qp
=

Bo Friis Nielsen

p2
0

0
p2

Discrete Time Markov Chains, Definition and classification

Small example

1 p

q
P =
0
0

(2)

p
0
q
0

0
0
p
0
0
p
q 1q

(1 p)2 + pq (1 p)p

q(1 p)
2qp
=
2
q
0

Bo Friis Nielsen

p2
0
2qp

0
p2
p(1 q)

Discrete Time Markov Chains, Definition and classification

Small example

1 p

q
P =
0
0

P (2)

p
0
q
0

0
0
p
0
0
p
q 1q

(1 p)2 + pq (1 p)p
p2
0

2

q(1

p)
2qp
0
p
=
2
q
0
2qp
p(1 q)

2

0
q
(1 q)q (1 q)2 + qp

Bo Friis Nielsen

Bo Friis Nielsen

Bo Friis Nielsen

n+m

Bo Friis Nielsen

n+m

Bo Friis Nielsen

n+m

Bo Friis Nielsen

n+m

P (B) =

Bo Friis Nielsen

n+m

At some intermediate time n we must be in some state k

We applyP
the law of total probability
P (B) = k P (B|Ak )

Bo Friis Nielsen

n+m

At some intermediate time n we must be in some state k

We applyP
the law of total probability
P (B) = k P (B|Ak ) P (Ak )
P (Xn+m = j|X0 = i)

Bo Friis Nielsen

n+m

At some intermediate time n we must be in some state k

We applyP
the law of total probability
P (B) = k P (B|Ak ) P (Ak )
P (Xn+m = j|X0 = i)
=

P (Xn+m = j|X0 = i, Xn = k )

Bo Friis Nielsen

n+m

At some intermediate time n we must be in some state k

We applyP
the law of total probability
P (B) = k P (B|Ak ) P (Ak )
P (Xn+m = j|X0 = i)
=

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

P (Xn+m = j|X0 = i, Xn = k )

Bo Friis Nielsen

Bo Friis Nielsen

by the Markov property we get

X
P (Xn+m = j|Xn = k )
k

Bo Friis Nielsen

by the Markov property we get

X
P (Xn+m = j|Xn = k ) P (Xn = k |X0 = i)
k

Bo Friis Nielsen

by the Markov property we get

X
P (Xn+m = j|Xn = k ) P (Xn = k |X0 = i)
k

(m)

(n)

Pkj Pik =

Bo Friis Nielsen

by the Markov property we get

X
P (Xn+m = j|Xn = k ) P (Xn = k |X0 = i)
k

(m)

(n)

Pkj Pik =

Bo Friis Nielsen

(n)

(m)

Pik Pkj

by the Markov property we get

X
P (Xn+m = j|Xn = k ) P (Xn = k |X0 = i)
k

(m)

(n)

Pkj Pik =

(n)

(m)

Pik Pkj

Bo Friis Nielsen

by the Markov property we get

X
P (Xn+m = j|Xn = k ) P (Xn = k |X0 = i)
k

(m)

(n)

Pkj Pik =

(n)

(m)

Pik Pkj

which in matrix formulation is

P (n+m) = P (n) P (m)

Bo Friis Nielsen

by the Markov property we get

X
P (Xn+m = j|Xn = k ) P (Xn = k |X0 = i)
k

(m)

(n)

Pkj Pik =

(n)

(m)

Pik Pkj

which in matrix formulation is

P (n+m) = P (n) P (m) = P n+m

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The probability of Xn

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The probability of Xn

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The probability of Xn

(n)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The probability of Xn

The behaviour conditional on X0 = i is known (Pij )

Define P (Xn = j)

(n)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The probability of Xn

The behaviour conditional on X0 = i is known (Pij )

Define P (Xn = j) = pj

(n)

(n)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The probability of Xn

The behaviour conditional on X0 = i is known (Pij )

Define P (Xn = j) = pj

(n)

(n)

(n)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The probability of Xn

The behaviour conditional on X0 = i is known (Pij )

Define P (Xn = j) = pj

(n)

(n)

(n)

p (n)

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The probability of Xn

The behaviour conditional on X0 = i is known (Pij )

Define P (Xn = j) = pj

(n)

(n)

(n)

p (n) = p

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

The probability of Xn

The behaviour conditional on X0 = i is known (Pij )

Define P (Xn = j) = pj

with p (n) = {pj } we find

(n)

(n)

(n)

p (n) = pP (n) = pP n

Bo Friis Nielsen

1 p

q
P =
0
0

p
0
q
0

Bo Friis Nielsen

0
0
p
0
0
p
q 1q

Small example - revisited

1 p

q
P =
0
0

with p = 13 , 0, 0, 23

p
0
q
0

Bo Friis Nielsen

0
0
p
0
0
p
q 1q

Small example - revisited

1 p p

q
0
P =
q
0
0
0

with p = 13 , 0, 0, 23 we get

0
0
p
0
0
p
q 1q

p (1)

Bo Friis Nielsen

Small example - revisited

1 p p

q
0
P =
q
0
0
0

with p = 13 , 0, 0, 23 we get

(1)


=

1
2
, 0, 0,
3
3

0
0
p
0
0
p
q 1q

Bo Friis Nielsen

Small example - revisited

1 p p

q
0
P =
q
0
0
0

with p = 13 , 0, 0, 23 we get

p (1) =

 1 p
1
2 q
, 0, 0,
3
3 0
0

p
0
q
0

0
0
p
0
0
p
q 1q

0
0
p
0
0
p
q 1q

Bo Friis Nielsen

Small example - revisited

1 p p

q
0
P =
q
0
0
0

with p = 13 , 0, 0, 23 we get

p (1) =

 1 p
1
2 q
, 0, 0,
3
3 0
0

p
0
q
0

0
0
p
0
0
p
q 1q

0
0
p
0
0
p
q 1q

Bo Friis Nielsen



= 1 p ,

3

Small example - revisited

1 p p

q
0
P =
q
0
0
0

with p = 13 , 0, 0, 23 we get

p (1) =

 1 p
1
2 q
, 0, 0,
3
3 0
0

p
0
q
0

0
0
p
0
0
p
q 1q

0
0
p
0
0
p
q 1q

Bo Friis Nielsen



= 1 p , p ,

3
3

Small example - revisited

1 p p

q
0
P =
q
0
0
0

with p = 13 , 0, 0, 23 we get

p (1) =

 1 p
1
2 q
, 0, 0,
3
3 0
0

p
0
q
0

0
0
p
0
0
p
q 1q

0
0
p
0
0
p
q 1q

Bo Friis Nielsen



= 1 p , p , 2q ,

3
3 3

Small example - revisited

1 p p

q
0
P =
q
0
0
0

with p = 13 , 0, 0, 23 we get

p (1) =

 1 p
1
2 q
, 0, 0,
3
3 0
0

p
0
q
0

0
0
p
0
0
p
q 1q

0
0
p
0
0
p
q 1q

Bo Friis Nielsen




= 1 p , p , 2q , 2(1 q)

3
3 3
3

Discrete Time Markov Chains, Definition and classification


p=

2
1
, 0, 0,
3
3


,

(1 p)2 + pq (1 p)p
p2
0

2

q(1

p)
2qp
0
p
P 2 =
q2
0
2qp
p(1 q)

0
q2
(1 q)q (1 q)2 + qp

Bo Friis Nielsen

p (2)

Bo Friis Nielsen

(2)


=

Bo Friis Nielsen

1
2
, 0, 0,
3
3

Discrete Time Markov Chains, Definition and classification

(2)


=

1
2
, 0, 0,
3
3

(1 p)2 + pq (1 p)p
p2
0

q(1 p)
2qp
0
p2

2

q
0
2qp
p(1 q)

2

0
q
(1 q)q (1 q)2 + qp

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

(2)


=

1
2
, 0, 0,
3
3

(1 p)2 + pq (1 p)p
p2
0

q(1 p)
2qp
0
p2

2

q
0
2qp
p(1 q)

2

0
q
(1 q)q (1 q)2 + qp


=

(1 p)2 + pq
,
3

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

(2)


=

1
2
, 0, 0,
3
3

(1 p)2 + pq (1 p)p
p2
0

q(1 p)
2qp
0
p2

2

q
0
2qp
p(1 q)

2

0
q
(1 q)q (1 q)2 + qp


=

(1 p)2 + pq (1 p)p
,
,
3
3

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

(2)


=

1
2
, 0, 0,
3
3

(1 p)2 + pq (1 p)p
p2
0

q(1 p)
2qp
0
p2

2

q
0
2qp
p(1 q)

2

0
q
(1 q)q (1 q)2 + qp


=

,
,
,
3
3
3

Bo Friis Nielsen

Discrete Time Markov Chains, Definition and classification

(2)


=

1
2
, 0, 0,
3
3

(1 p)2 + pq (1 p)p
p2
0

q(1 p)
2qp
0
p2

2

q
0
2qp
p(1 q)

2

0
q
(1 q)q (1 q)2 + qp


=

,
,
,
3
3
3
3

Bo Friis Nielsen

Consider the transition probability matrix

1 0 0

P =
0 0 1
Define
T = min {n 0 : Xn = 0 or Xn = 2}
and
u = P(XT = 0|X0 = 1)

Bo Friis Nielsen

v = E(T |X0 = 1)

First step analysis - absorption probability

u = P(XT = 0|X0 = 1)
=

2
X

k =0

2
X

k =0

And we find
u=

P1,0

=
1 P1,1
+

Bo Friis Nielsen

= E(T |X0 = 1)
=

2
X

k =0

= 1+

2
X

k =0

And we find
v=

1
1
=
1 P1,1
1

Bo Friis Nielsen

More than one transient state

1
0
0
0

P1,0 P1,1 P1,2 P1,3
P =
P2,0 P2,1 P2,2 P2,3
0
0
0
1

Here we will need conditional probabilities

ui = P(XT = 0|X0 = i)

Bo Friis Nielsen

u1 = P1,0 + P1,1 u1 + P1,2 u2

u2 = P2,0 + P2,1 u1 + P2,2 u2
and
v1 = 1 + P1,1 v1 + P1,2 v2
v2 = 1 + P2,1 v1 + P2,2 v2

Bo Friis Nielsen

Q R

P =
0 I

Bo Friis Nielsen

General absorbing Markov chain

T = min {n 0, Xn r }
In state j we accumulate reward g(j), wi is expected total
reward conditioned on start in state i
!
T
1
X
wi = E
g(Xn )|X0 = i
n=0

wi = g(i) +

Pi,j wj

Bo Friis Nielsen

Bo Friis Nielsen

Bo Friis Nielsen

Bo Friis Nielsen