Lec 6 Mean Time in Transient States

San José State University
Math 263: Stochastic Processes
Mean time spent in transient states
Dr. Guangliang Chen

This lecture is based on the following textbook sections:
• Section 4.6
• Section 4.5.1
Outline of the presentation
• Mean time spend in transient states
• Transition probability between transient states
• The Gambler’s Ruin problem

Math 263, Mean time spent in transient states
Mean time spent in transient states

Consider a finite-state Markov chain with transient states numbered as
T = {1, . . . , t } (and recurrent states numbered above t or under 1).
For example, in the Gambler’s Ruin problem, let X n denote the gambler’s
fortune after the n th bet. Then {X n , n = 0, 1, 2, . . .} is a Markov chain:
1 1
p p p p
0 1 2 N
1−p 1−p 1−p 1−p
The transient states are 1, . . . , t = N − 1 (and the recurrent states are 0, N ).
Dr. Guangliang Chen | Mathematics & Statistics, San José State University 3/17
For any two transient states i , j ∈ T , let s i j denote the expected number
of time periods that the Markov chain is in state j , given that it starts in
state i :
s i j = E (T0 | X 0 = i ) ,
where ∞
X
T0 = In , I n = 1X n = j
n=0
The following theorem shows how to compute all the s i j collectively.

Theorem 0.1. Let PT = (p i j )1≤i , j ≤t ∈ Rt ×t , the transition matrix restricted
to the transient states, and S = (s i j ) ∈ Rt ×t , the matrix of mean times in
transient states (when starting in transient states). Then
S = (I − PT )−1 .
Proof. We condition on the initial transition:
s i j = E (T0 | X 0 = i )
X
= E(T0 | X 0 = i , X 1 = k)P (X 1 = k | X 0 = i )
k
(δi j + s k j )p i k = δi j +
X X
= p i k sk j
k k
t
= δi j +
X
p i k sk j (s k j = 0 for recurrent states k)
k=1
In matrix notation, this equation is
S = I + PT S −→ (I − PT )S = I
From this we obtain that S = (I − PT )−1 .
Example 0.1 (Gamber’s Ruin). Suppose p = 12 , N = 5. Then the transient

states are T = {1, 2, 3, 4}, and
   
0.5 1.6 1.2 0.8 0.4
   
0.5 0.5  1.2 2.4 1.6 0.8
PT =   −→ S= 

 0.5 0.5

0.8 1.6 2.4 1.2
 
0.5 0.4 0.8 1.2 1.6
1 1
p p p p
0 1 2 N
1−p 1−p 1−p 1−p
Transition probability between two states
Def 0.1. For any two states i , j , define by f i j the probability that starting
in state i , the process will ever make a transition into state j :
f i j = P ∪∞
¡ ¢
n=1 {X n = j } | X 0 = i
Remark. Compare with f i i and f i(n)

i
.
Theorem 0.2. For any two transient states i , j ,
s i j − δi j
fi j =
sj j
Proof. It follows from the following equation:
s i j = δi j + f i j · s j j + 1 − f i j · 0
¡ ¢
Example 0.2 (Cont’d). Starting the dollar amounts 1, 2, 3, 4, the probabil-

ities of the gambler ever reaching each of those amounts (again) are given
by
   
1.6 1.2 0.8 0.4 0.3750 0.5000 0.3333 0.2500
   
1.2 2.4 1.6 0.8 0.7500 0.5833 0.6667 0.5000
S=
  −→ FT = 
 0.8 1.6 2.4 1.2 

0.5000
 0.6667 0.5833 0.7500
0.4 0.8 1.2 1.6 0.2500 0.3333 0.5000 0.3750
1 1
p p p p
0 1 2 N
1−p 1−p 1−p 1−p
From transient to recurrent

Example 0.3. Consider a gambler who at each play of the game has
probability p of winning one unit and probability q = 1 − p of losing one
unit. Assuming that successive plays of the game are independent, what is
the probability that, starting with i units, the gambler’s fortune will reach
N before reaching 0?
1 1
p p p p
0 1 2 N
1−p 1−p 1−p 1−p
Solution. Let p i = f i N for i = 1, . . . , N − 1 and q = 1 − p for convenience.
By conditioning on X 1 we get that
p i = p · p i +1 + q · p i −1 , i = 1, . . . , N − 1
where we have defined p 0 = 0, p N = 1.
Write p i = (p + q) · p i and substitute it into the above recursive relation to

get that
p i +1 − p i q
q(p i − p i −1 ) = p(p i +1 − p i ) −→ =
p i − p i −1 p
It follows that
µ ¶i −1
q
//0 )
p i − p i −1 = (p 1 − p , i = 1, . . . , N
p
and by telescoping,
p i = (p i − p i −1 ) + (p i −1 − p i −2 ) + · · · + (p 2 − p 1 ) + (p 1 − p 0 )
µ ¶i −1 µ ¶i −2
q q q
µ ¶
= p1 + p1 + · · · + p1 + p1
p p p
 i
p · 1−(q/p) , p 6= q
1 1−(q/p)
=
p · i , p=q
1
To determine p 1 , use p N = 1:

1−(q/p)

1−(q/p)N
, p 6= q
p1 =
1, p=q
N
Consequently,
 i
 1−(q/p) , p 6= q
1−(q/p)N
pi =
i , p=q
N
Example 0.4 (Gambler’s Ruin, cont’d). , Assume the same setting as

before. If the player quits gambling once he either reaches a fortune of N
or goes broke (whatever comes first), how long on average will that take?
Solution. Let
Ti = min{n ≥ 0 : X n = 0 or X n = N | X 0 = i }, i = 1, . . . , N − 1
We would like to find mi = E(Ti ).
By conditioning on X 1 we get that
m i = E(Ti | X 1 = i + 1)P (X 1 = i + 1 | X 0 = i )
+ E(Ti | X 1 = i − 1)P (X 1 = i − 1 | X 0 = i )
= (1 + E(Ti +1 )) · p + (1 + E(Ti −1 )) · q
= 1 + p · m i +1 + q · m i −1 , i = 1, . . . , N − 1
We solve the above recursive relations, along with boundary conditions

T0 = T N = 0, only for the case of p = q = 1/2:
m i = i · (N − i ), i = 0, 1, . . . , N
Remark. When p 6= q , it can be shown that

N 1 − (q/p)i i
· ¸
mi = · − , i = 0, 1, . . . , N
p − q 1 − (q/p)N N

Lec 6 Mean Time in Transient States

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Lec 6 Mean Time in Transient States

Uploaded by

Copyright:

Available Formats

San José State University

Math 263: Stochastic Processes

Mean time spent in transient states

Dr. Guangliang Chen

Outline of the presentation

• Mean time spend in transient states

• Transition probability between transient states

• The Gambler’s Ruin problem

Mean time spent in transient states

The transient states are 1, . . . , t = N − 1 (and the recurrent states are 0, N ).

The following theorem shows how to compute all the s i j collectively.

Proof. We condition on the initial transition:

In matrix notation, this equation is

From this we obtain that S = (I − PT )−1 .

Example 0.1 (Gamber’s Ruin). Suppose p = 12 , N = 5. Then the transient

Transition probability between two states

Remark. Compare with f i i and f i(n)

Theorem 0.2. For any two transient states i , j ,

Proof. It follows from the following equation:

Example 0.2 (Cont’d). Starting the dollar amounts 1, 2, 3, 4, the probabil-

From transient to recurrent

Solution. Let p i = f i N for i = 1, . . . , N − 1 and q = 1 − p for convenience.

By conditioning on X 1 we get that

where we have defined p 0 = 0, p N = 1.

Write p i = (p + q) · p i and substitute it into the above recursive relation to

Example 0.4 (Gambler’s Ruin, cont’d). , Assume the same setting as

We would like to find mi = E(Ti ).

By conditioning on X 1 we get that

We solve the above recursive relations, along with boundary conditions

Remark. When p 6= q , it can be shown that

You might also like