STOCHASTIC PROCESSES 1: HSTS416

ASSIGNMENT1

1. For each of the following Markov chains, determine whether they are transient or
recurrent.

0 1/3 1/3 1/3

1/3 0 1/3 1/3
(a) 𝑷 = ‖ ‖
1/3 1/3 0 1/3
1/3 1/3 1/3 0

1/4 3/4 0 0 0
1/2 1/2 0 0 0
(b) 𝑃 = ‖
‖ 0 0 1 0 0‖‖
0 0 1/3 2/3 0
1 0 0 0 0

2. State and prove the Chapman-Kolmogorov equations.

3. (a) If 𝑖 ↔ 𝑗 and 𝑗 ↔ 𝑘, prove that 𝑖 ↔ 𝑘.

(b) If 𝑖 ↔ 𝑗, prove that 𝑑(𝑖) = 𝑑(𝑗).

4. Consider the Markov chain with three states 0, 1 and 2 and a transition matrix:

𝟐/𝟓 𝟐/𝟓 𝟏/𝟓

𝑷 = ‖𝟏/𝟓 𝟑/𝟓 𝟏/𝟓‖
𝟏/𝟓 𝟐/𝟓 𝟐/𝟓

(a) Explain what is meant by the statement that “a Markov chain is an irreducible
recurrent chain”.
(b) Show, stating any genera results that you may assume, that this statement is true
for the present chain.
(c) Find the stationary distribution for this chain.

5. A Markov Chain for the progression of a disease, 𝑋𝑛 , denotes the level of severity in
year 𝑛, for 𝑛 = 0, 1, 2, …. The state space is {1, 2, 3, 4} with the following interpretations:
state 1 the symptoms are under control, states 2 and 3 represent moderate and severe
symptoms respectively, while state 4 represents a permanent disability. The transition
probability matrix is:

1/2 1/4 0 1/4

0 1/2 1/4 1/4
𝑷=‖ ‖
0 0 1/2 1/2
0 0 0 1
(a) Classify the four states as transient or recurrent giving reasons.
(b) What does this tell you about the long-run fate of someone with this disease?
(c) Calculate the 2-step transition matrix.
(d) State the probability that a patient whose symptoms are:
 (i) Moderate will be permanently disabled one year later.
(ii) Under control will have severe symptoms two years later.
(e) Calculate the probability that a patient whose symptoms are moderate will have
severe symptoms four years later.
A new treatment becomes available but only to permanently disabled patients, all
whom receive treatment. This has a 50% success rate in which a patient returns to
state 1 and is subject to the same transition probabilities as before. A patient whose
treatment is unsuccessful remains in state 4 receiving a further round of treatment the
following year.
(f) Write the new transition probability matrix for this new Markov chain and classify
the states as transient or recurrent.
(g) Calculate the stationary distribution of the new chain. The population from the
disease currently has no members receiving the new treatment. What proportion
of the population will receive treatment in the long run?
(h) The annual cost of health care for patients is zero in state 1, $k in state 2, $2k in
state 3 and $4k in state 4. Calculate the expected annual cost per patient when the
system is in steady state.

6. Consider the following finite-state Markov chain.

(a) Identify all the classes present in the chain and the states belonging to each
class.
(b) Find the period of each class.
(c) Determine whether the class is transient or recurrent.
END OF ASSIGNMENT