Markov Chain

Question 1
An engineering professor purchases a new computer every two years
with preferences for the three model: M1, M2, and M3.
If the present model is M1, the next computer may be M2 with
probability 0.2, or M3 with probability 0.15.
If present model is M2, the probabilities of switching to M1 and M3
are 0.6 and 0.25, respectively.
And, if the present model is M3, then the probabilities of switching to
M1 and M2 are 0.5 and 0.1, respectively.
Represent the situation as a Markov chain and determine the
probability that the professor will purchase the current model in four
years
States:
1. M1
2. M2
3. M3

The Markov Chain is:

The probability that the professor will purchase the current model in 4
years. P (2-step)

𝑃 𝑀1 𝑀1 = 0.6175

𝑃 𝑀2 𝑀2 = 0.1675

𝑃 𝑀3 𝑀3 = 0.26
Question 2
When I borrow a book from the city library, I usually try to return it
after one week. Depending on the length of the book and my free time,
there is a 30% chance that I may keep it for another week. If I have had
the book for two weeks, there is a 10% chance that I will keep it for an
additional week. Under no condition do I keep it for more than three
weeks.
• Express the situation as a Markov chain.
• Determine the average number of weeks I keep a book before
returning it to the library.
States:
1. 1: 1 Week
2. 2: 2 Weeks
3. 3: 3 Weeks
4. L: Library
The Markov Chain is:
b) Determine the average number of weeks before returning a book to
the library is 𝐼 − 𝑁 −1 ∗ 1
I will keep the book 1.33 week on an average
Question 3
Population dynamics is impacted by the continual movement of people
who are seeking better quality of life or better employment. The city of
Mobile has an inner city population, a suburban population, and a
surrounding rural population. The census taken in 10-year intervals
shows that 10% of the rural population move to the suburbs and 5% to
the inner city. For the suburban population, 30% move to rural areas
and 15% to the inner city. Inner-city population would not move into
suburbs, but 20% of them move to the quiet rural life.
• Express the population dynamics as a Markov chain.
• If the greater Mobile area currently includes 20,000 rural residents,
100,000 suburbanites, and 30,000 inner city inhabitants, what will the
population distribution be in 10 years? In 20 years?
• Determine the long-run population picture of Mobile.
States:
1. Inner city population
2. Suburban population
3. Surrounding rural population

a). The Markov Chain is:

b). Initial Probabilities
0 30000 100000 20000
𝑎 = 150000 150000 150000
𝑎 0 = 0.2 0.67 0.13

Population distribution in 10 years

Population = 150000 * P[1-step] = 150000 * a0 * P
State Absolute [1-step] Population
Inner City 0.267 40050
Suburban 0.387 57225
Surrounding rural 0.352 52725
Population distribution in 20 years
Population = 150000 * P[2-step] = 150000 * a0 * P2

State Absolute [2-step] Population

Inner City 0.288 43200
Suburban 0.245 36750
Surrounding rural 0.467 70150
c). Long run population = 150000 * 𝜋𝑖
𝜋1 𝜋2 𝜋3 = 𝜋1 𝜋2 𝜋3 ∗ 𝑃

𝜋1 + 𝜋2 + 𝜋3 = 1

𝜋1 = 0.254238, 𝜋2 = 0.135593, 𝜋3 = 0.61017

c). Long run population

𝜋1 = 0.254238, 𝜋2 = 0.135593, 𝜋3 = 0.61017

State Steady State Population

Inner City 0.254238 0.254238*150,000
= 38135
Suburban 0.135593 0.135593*150,000
= 20339
Surrounding rural 0.61017 0.61017*150,000
= 91526
Question 4
Some ex-cons spend the rest of their lives in one four of states: free, on
trial, in jail, or on probation. At the start of each year, statistics show
that there is 50% chance that a free ex-con will commit a new crime
and go on trial. The judge may send the ex-con to jail with probability
0.6 or grant probation with probability 0.4. Once in jail, 10% of ex-cons
will be set free for good behavior. Of those who are on probation, 10%
commit new crimes and are arraigned for new trials, 50% will go back
to finish their sentence for violating probation orders, and 10% will be
set free for lack of evidence. Taxpayers underwrite the costs associated
with the punishment of the ex-felons. It is estimated that a trial will
cost about $5,000, an average jail sentence will cost $20,000, and an
average probation period will cost $2,000.
• Determine the expected cost per ex-con.
• How often does an ex-con return to jail? Go on trial? Get set free?
States:
1. F: free
2. T: on trial
3. J: in jail
4. P: on probation

The Markov Chain is:

 𝜋1 𝜋2 𝜋3 𝜋4 = 𝜋1 𝜋2 𝜋3 𝜋4 ∗ 𝑃

The solution is
𝜋1 = 0.153123 𝜋2 = 0.081206 𝜋3 = 0.719257 𝜋4 = 0.046404

Expected cost per ex-con

E[cost/ex-con] = 0 ∗ 0.153123 + 5000 ∗ 0.081206
+20000 ∗ 0.719257 + 2000 ∗ 0.046404
= $ 14884
b). How often does an ex-con return to jail? Go on trial? Be set free?

1
𝜇𝐽𝐽 = = 1.39 𝑦𝑒𝑎𝑟𝑠
𝜋3

1
𝜇𝑇𝑇 = = 12.31 𝑦𝑒𝑎𝑟𝑠
𝜋2

1
𝜇𝐹𝐹 = = 6.53 𝑦𝑒𝑎𝑟𝑠
𝜋1
Question 5
Jim and Joe start a game with five tokens, three for Jim and two for Joe.
A coin is tossed and if the outcome is heads, Jim gives Joe a token; else,
Jim gets a token from Joe. The game ends when Jim or Joe has all the
tokens. At this point, there is 30% chance that Jim and Joe will continue
to play the game, again starting with three tokens for Jim and two for
Joe.
a. Represent the game as a Markov chain.
b. Determine the probability that Joe will win in three-coin tosses.
That Jim will win in three coin tosses.
c. Determine the probability that a game will end in Jim's favor. Joe's
favor.
d. Determine the average number of coin tosses needed before Jim
wins. Joe wins.
States:
(Jim-Joe)
3-2, 2-3, 1-4, 4-1, 0-5, 5-0
Initial Probabilities:
𝑎0 = 1 0 0 0 0 0
The Markov Chain is:
b. probability that Joe will win in three-coin tosses, and that Jim will win in
three coin tosses.

Output (3-step) transition matrix

P{Jim wins in 3 tosses} = P 3 − 2 5 − 0 = 0.175

P{Joe wins in 3 tosses} = P 3 − 2 0 − 5 = 0.125
c. probability that a game will end in Jim's favor and Joe's favor.

Steady State Probabilities:

𝜋1 𝜋2 𝜋3 𝜋4 𝜋5 𝜋6

𝜋1 = 0.257143 𝜋2 = 0.171429
𝜋3 = 0.085714 𝜋4 = 0.128571
𝜋5 = 0.142857 𝜋6 = 0.214286

P{game ends in Jim’s favor} = 𝜋6 = 0.214286

P{game ends in Joe’s favor} = 𝜋5 = 0.142857
d. average number of coin tosses needed before Jim wins, and Joe wins

average 12.2 number of coin tosses till Jim wins

average 20 number of coin tosses till Joe wins
Question 6
The registration committee of the IT department at AAA university has modeled the path
of the student through the department a Markov chain. The student in this department
can be in the first year, second year, or in he third year of studies. If he finished successfully
the third then he graduate. Any student can quit the department at any year of studies. We
give the following transition probabilities matrix.

a. What is the probability that the student will be in the third year after two years if he is
currently in the second year?
b. What is the steady state probability of the Q state?
c. What is the probability that the student will graduate from this school if he is currently
in the first year?
d. What is the probability that the student will graduate from this school in exactly three
years if he is currently in the first year?
e. What is the average number of years that the student may spend in this school?
a. What is the probability that the student will be in the third year after two
years if he is currently in the second year?

(2)
𝑃23 = 𝑃21 ∗ 𝑃13 + 𝑃22 ∗ 𝑃23 + 𝑃23 ∗ 𝑃33 + 𝑃24 ∗ 𝑃43 + 𝑃25 ∗ 𝑃53

(2)
𝑃23 = 0 ∗ 0 + 0.3 ∗ 0.5 + 0.5 ∗ 0.2 + 0.2 ∗ 0 + 0 ∗ 0

(2)
𝑃23 = 0.25

b. What is the steady state probability of Q State?

This is not an ergodic Markov chain, So we cannot compute the steady State
probabilities.
c. What is the probability that the student will graduate from this
school if he is currently in the first year?
𝑃𝑟𝑜𝑏𝑎𝑏𝑖𝑡𝑖𝑒𝑠 𝑜𝑓 𝑎𝑏𝑠𝑜𝑟𝑝𝑡𝑖𝑜𝑛 = 𝐼 − 𝑁 −1 ∗ 𝐴

probability that the student will graduate from this school if he is

currently in the first year is 0.55
d. The probability that the student will graduate from this school in exactly three years.
P(3-step)

𝑎0 ∗ 𝑃 3 = 0.01 0.13 0.25 0.37 0.25

So, The probability that the student will graduate from this school in exactly three years is
0.25
e. What is the average number of years that the student may spend in
this school?

the average number of years that the student may spend in this school
is 3.28 years