University of Pretoria

WST312 Stochastic Processes

Past Questions Memo

Chapter 3
Question 1 Suppose that a person possesses three umbrellas. If it rains in the morning, the
person takes an umbrella (if there are any at home) to the office. If it rains in the afternoon the
person takes an umbrella home (if there are any at the office). The probability that it will rain
in the morning and the probability it will rain in the afternoon are both p and are independent
events. Whether or not it rains on a particular day is independent from what happens on any
other day. Let X t be the number of umbrellas at home at the end of the t th day.
a) Explain why { X t : t  1,2,3,...} is a time-homogeneous irreducible Markov Chain.

b) Determine the transition probability matrix for the process.

c) Suppose the transition probability matrix in (b) is given by

1
  0.5 0.5 0 0 
0.25 0.5 0.25 0 
P .
 0 0.25 0.5 0.25
 
 0 0 0.25 0.75
(i) Determine the limiting probabilities for the process.

(ii) What percentage of the time will the person get wet either in the morning or in the evening?

Question 2 A time-homogeneous Markov Chain X t : t  T  with state space S  {1,2,3,4,5}

has the following transition probability matrix,
0.2 0.3 0.2 0.2 0.1
 0 0.3 0 0.4 0.3
 
P  0.2 0 0.7 0 0.1 .
 
0 0 0 0.8 0.2
 0 0.3 0 0.7 0 
a) Prove the Chapman-Kolmogorov equations for this process.
Theory
b) Calculate P[ X 10  3 | X 8  3] stating the result you make use of.

2
c) Calculate lim P[ X n  4 | X 3  1] .
n 

d) Calculate P[ R10  7 | X 10  4] where Rt is the time you have to wait before leaving the
state occupied at time t .

Question 3 Consider people who may consider enrolling in a university and obtaining a
degree 3 years later. Let a person be in one of 6 states: State 1 – doesn’t have a degree and will
not enroll, State 2 – doesn’t have a degree but may enroll, State 3 – first year student, State 4
– second year student, State 5 – third year student, State 6 – has a degree. The transition
probability matrix for this process is given by the following,
 1 0 0 0 0 0 
 0.7 0.2 0.1 0 0 0 

0.05 0 0.1 0.85 0 0 
P 
0.07 0 0 0.13 0.8 0 
 0.1 0 0 0 0.15 0.75
 
 0 0 0 0 0 1 
a) If the university only allows a student to attempt any one year of his/her degree twice (but
has the option of an alternative study field the following year), what is the probability a first
year student will not be allowed to continue studying his/her present degree?

3
 P 0
b) Explain the form P *   1
Q 
of the transition probability matrix P .
R

c) Define the probability g ij(n) and prove that G ( n )  [ g ij( n ) ]  Q n1 R .

d) What is the probability that a first year student completes his/her degree in exactly 3 years?

e) What is the expected number of years that it will take a student currently in first year to
either get a degree or quit studying altogether?

4
You may find the following useful:
 1 0 0 0 0 0 
 0.845 0.04 0.03 0.085 0 0 

0.1145 0 0.01 0.1955 0.685 0 
P2   
 0.1591 0 0 0.0169 0.224 0.6 
 0.115 0 0 0 0.0225 0.8625 
 
 0 0 0 0 0 1 
 1 0 0 0 0 0 
 0.88045 0.008 0.007 0.03655 0.068 0 
 
 0. 196685 0 0.001 0 .033915 0 .2584 0 .51 
P3   ,
 0 . 182683 0 0 0. 002197 0 .04712 0 .768 
 0.11725 0 0 0 0.003375 0.879375 
 
 0 0 0 0 0 1 
1
 1 0 0 0 0.2 0.1 0 0  1.25 0.1388889 0.135696 0.1277139 
   
 0 1 0 0  0 0.1 0.85 0    0 1.1111111 1.0855683 1.0217114 
 0   
0 1 0  0 0 0.13 0.8    0 0 1.1494253 1.081812 
     
 0 0.15 
 0 0 1  0 0 0  0 0 0 1.1764706 

2
0.2 0.1 0 0  0.04 0.03 0.085 0 
 0 0.1 0.85 0   0 0.01 0.1955 0.68 
 
0 0 0.13 0.8   0 0 0.0169 0.224 
   
0 0 0 0.15  0 0 0 0.225

Question 4 At the end of each financial year an independent organization ranks the
performance of the unit trusts in a particular sector and classifies them into four classes (class
1 being the best performer). Past experience has shown that at the end of each financial year a
fund will either remain in the same class (with probability 1  2 ) or move to a neighbouring
class (with equal probability for each possible neighbouring move). However, if a fund has
remained in the top or bottom class for 2 consecutive years, it has probability
1   ( where    ) of remaining in that same class the following year.

5
a) Construct a Markov Chain with six states to model this situation clearly defining your chosen
states.

b) Write down the transition probability matrix for your process in (a).

Question 5 Consider a two-state time-homogeneous Markov chain with transition

probability matrix
 a 1  a
P
b 
.
1  b
 p (n) (n)
p 01  2b  a  1 1  b
Prove that if P n   00( n ) (n) 
then p11( n )   (1  a  b) n .
 p10 p11  ab ab
Theory

Question 6 Consider a NCD system with discount levels 0%, 10%, 20 % and 30% and with
the following set of rules for moving between the states:
- One claim free year results in a move up one level to a higher discount level (or to stay in
the maximum discount category)
- Two claim free years in a row results in a move up to the highest discount level.
- If a claim is made in any given year the policyholder moves down one discount level (or
stays in 0% discount).
- If a policyholder makes claims in two consecutive years the policyholder moves to 0%
discount.
- Assume only a single claim can be made in a year.
(a) Determine the 4-state transition probability matrix for this process if the probability of
claiming in any given year is p .

6
(b) Is the process a Markov process? Explain fully.

Question 7 Consider the following transition probability matrix, for a time-homogeneous

Markov chain { X n : n  0,1,2,3,...} and with S  {1,2,3,4,5} :
 0. 2 0.1 0.5
0. 2 0
0
1 0 0 0 

P   0. 3
0.4 0.1 0.2 0 
 
0
0.4 0 0.3 0.3
0.2
0.1 0 0.1 0.6
Find the limiting distribution for the process given.
n n
(n)
State 2 is absorbing (1/2 MARK) therefore p 22 1 and pi(2n )  0 for i  1,3,4,5. (1 MARK)
 

The are two remaining equivalence classes: C 2  {3} and C3  {1,4,5} . Both these classes
n
 0 for j  1,3,4,5 and for any i (1 MARK).
( n)
are transient (1/2 MARK), thus pij


Question 8 Consider a time-homogenous Markov Chain {X n : n  0,1,2,...} and suppose we

consider 2 cases (1) S  {1,2,3,..., m} and (2) S  {1,2,3,...} . We are interested in the limiting
distribution for this process. Make a detailed comparison of the two cases with respect to the
following:
CASE 1: S  {1,2,3,..., m} CASE 2: S  {1,2,3,...}
(a) The assumptions made about the process for the limiting distribution to exist.

The process is irreducible, aperiodic and finite The process is irreducible, recurrent
(1 MARK) (or ergodic) and aperiodic (1 MARK)
(b) The equations used to solve for the limiting probabilities


 P     j    i pij for j  0,1,2,... i.e.

i 0
 1  1 (1 MARK) 
    P and 
i 0
i  1 .(1 MARK)

(c) Properties of the limiting probabilities

 
  In an equivalence class C , if there
  exists a state i such that
lim P n      where    1 ,  2 ,..., m 
n 
  lim pii   i  0 ,
(n)
then
n 

   j  lim p (jjn )  0 for all j  C .
n 

  is stationary and  i  0,   i  1
m
with 0   j  1and   j  1.
j 1 i 0

and   is the stationary distribution (1 1/2 MARKS)

7
 (1 1/2 MARKS)

(d) The uniqueness of the limiting probabilities.

  is unique (1/2 MARK)   is unique (1/2 MARK)

(e) The form of the limiting probabilities i.e. a formula for them.
  1
   if  i  
lim p (n)
  i
  n 
ii

lim P n      where    1 ,  2 ,..., m  0 if  i  


n 
 
 lim p ji  lim pii if j  i
(n) (n)
n  n 
  

(1 MARK) where  i   nf ii( n ) (1 MARK)

n 0

Question 9 a) Consider a time-homogeneous Markov chain with transition probability matrix

1  a a 
P  . Explain the range of values that a and b can take on which result in this
 b 1  b
being a valid Markov chain which is:
(i) irreducible

(ii) aperiodic.

b) For the statements that follow consider i  C1 and j , k  C 2 , for two equivalence classes C1
and C 2 . Each statement in incorrect in some way. Correct each statement to reflect the correct
result.
(i) i and j communicate and have the same period.

(ii) pij( m)  0 for some m and p (jin )  0 for some n .

8
 (iii) If d k  1 and p (jkn )  0 then p (jknm)  0 for all m .

Question 10 For a finite, aperiodic Markov Chain which is not irreducible, describe how you
would determine the limiting distribution. Make use of the transition probability matrix and
discuss all possibilities which can occur in practice.

Question 11 A motor insurer operates a no claims discount system with the following levels
of discount: {0%, 25%, 50%, 60%}. The rules governing a policyholder’s discount level, based
upon the number of claims made in the previous year, are as follows:
- Following a year with no claims, the policyholder moves up one discount level, or
remains at the 60% level.
- Following a year with one claim, the policyholder moves down one discount level,
or remains at 0% level.
- Following a year with two or more claims, the policyholder moves down two
discount levels (subject to a limit of the 0% discount level).
The number of claims made by a policyholder in a year is assumed to follow a Poisson
distribution with mean 0.3.
a) Determine the transition matrix for this NCD system.

b) Calculate the stationary distribution  of the system.

9
 c) Calculate the expected average long term level of discount.

d) Explain how you would find the distribution of the recurrence time for the top discount level
for n transitions.

The following data shows the number of the insurer’s 130200 policyholders in the portfolio
classified by the number of claims each policyholder made in the last year. This information
was used to estimate the mean of 0.3.
No Claims 96632
One Claim 28648

10
 Two Claims 4400
Three Claims 476
Four Claims 36
Five Claims 8

e) Explain how you would test if this data was in fact distributed Poisson with mean
0.3.

f) Complete the test in (e) to determine if the data is in fact Poisson with mean 0.3.
Degrees of freedom df  df2 ,0.05
1 3.841
2 5.991
3 7.815
4 9.488
5 11.070

g) How does the result in (f) affect the average level of discount applied?

Question 12 a) Explain what is meant by a stationary stochastic process.

A stochastic process X t : t  T  is a stationary process if for any integer n , any values

t1  t 2  ...  t n which are all elements of T and any value k such that k  t1 , k  t 2 ,..., k  t n
are all elements of T , it is true that the joint distribution of X t1 , X t2 ,..., X tn is the same as
the joint distribution of X t1  k , X t2  k ,..., X tn  k .

b) Give an example of a process which is not stationary. Show that the process given is indeed
not stationary.

The random walk is not stationary:

11
  t 
If we let EX t     0 then E[Yt ]  E  X i   t which depends on t . Thus if the
 i 1 
expected values are not equal the distributions cannot be equal either.

Question 13 Consider a time-homogeneous Markov chain X t , t  0,1,2,... with state space

 0.1 0.4 0.5
S  {1,2,3} . If P 2  0.3 0.1 0.6 ,
0.5 0.3 0.2
a) provide a formula for the probability of the transition from state 2 to state 1 to state 3 in 4
( 2) ( 2)
steps, in terms of the 2-step transition probabilities. p21 p13
b) provide a formula for the probability of the transition from state 2 to state 3 in 4 steps, in
3
terms of the 2-step transition probabilities. p
k 1
( 2)
2k p k( 23)

c) Name and prove the result you used in (b). You may use the following result without proof:
P[ A  B | C ]  P[ A | C ]P[ B | A  C ] . Chapman-Kolmogorov equations (proof)
d) Provide numerical answers for (a) and (b).
a) (0.3)(0.5) = 0.15
b) 0.15+0.07+0.12 = 0.34

Question 14 Indicate whether the following statements are true or false and correct those
that are false.

Statement True/False Correction (if False)

(i) i  j, j  k  i  k True
(ii) If i  C1 and j  C1 and False If i  C1 and j  C 2 and pi(,mj)  0 for
pi(,mj)  0 for some m , then some m , then p (jn,k)  0 for all k  C1
p (jn,k)  0 for all k  C1 and for and for all n .
all n .
(iii) If i  j then i and j have False If i  j then i and j have the same
the same period. period.
(iv) State i is recurrent if and True

only if p
n 0
(n)
ii  .

(v) If state i is recurrent and if False If state i is recurrent and if i  j , then

i  j , then state j is recurrent. state j is recurrent.

Question 15 Suppose a two-state Markov chain with state space S  {0,1} has transition
1  a a  a (1  a  b) n
probability matrix P    . Show that p11  a  b  b a  b .
(n)
Proof
 b 1  b

Question 16 Consider a time-homogenous Markov chain { X t , t  0,1,2,...}.

a) Define fully the probabilities

12
 (i) f ij(n) is the probability, given that the process started in state i , that the process
will visit state j for the first time after n transitions.

(ii) g ij(n) For i  T and j  T C , g ij(n) is the probability, given that the process begins
in state i  T , the process will visit only transient states for n  1 transitions and at the n th
transition goes to state j  T C .
(iii) hij(n ) For i  T and j  T , hij(n ) is the probability, given that the process started
in state i , that state j will be visited n times before the process visit some recurrent state.
b) Provide formulas to in order to calculate
n 1
(i) f ij(n) f ij( n )  pi(,nj)   f ij( k ) p (jjn  k )
k 1

P 
(ii) g ij(n) G ( n )  [ g ij( n ) ]  Q n1 R where P   1
R Q


(iii) mij if mij   n hij( n ) . M  ( I  Q) 1
n 1

Question 17 Consider a time-homogeneous Markov chain with the following transition

0.3 0.2 0 0 0.5 0 
 0 0.5 0.5 0 0 0 

 0 0. 7 0. 3 0 0 0
probability matrix: P    with that space S  {1,2,3,4,5,6} .
 0 0 0 . 8 0 0 0 .2 
0 0 0 0 1 0
 
 0 0 0 0.6 0 0.4
a) Give all the equivalence classes for this process. Explain fully.
C1  {5} : since p55( n)  1 n and p5( ni )  0 for i  1,2,3,4,6 and n state 5 is absorbing and
does not communicate with any other state. (1 mark)

C 2  {2,3} : since p23  0.5  0 and p32  0.7  0 states 2 and 3 communicate and thus
belong to one class since they both cannot move to any other state (1 mark)

C3  {1} : since pi1  0 for i  2,3,4,5,6 thus once the process leaves state 1 it can never
return and therefore doesn’t communicate with any other state. (1 mark)

C 4  {4,6} : since p46  0.2  0 and p64  0.6  0 states 4 and 6 communicate and thus
belong to one class (1 mark)

b) Determine the limiting probabilities for this process, explaining fully how you obtain your
answers. It is not necessary to give your answer as  .

For C1  {5} : lim p55( n )  1 and lim p5( ni )  0 for i  1,2,3,4,6 (1 mark)
n  n 

13
For C 2  {2,3} : this class is recurrent since the process will remain in this class once it gets
there and in a finite irreducible process all states must be recurrent.(1 mark) Thus we solve
 2  3 
0.5 0.5
   2  3  and  2   3  1 (1 mark).
0 .7 0.3 
0.5
0.5 2  0.7 3   2  0.7 3  0.5 2   3  2
0.7
and  2   3  1
0.5
So  2  2 1 (2 marks)
0.7
7
2 
12
5
and  1 
12

For C3  {1} : this class is transient since we can leave the class and go to state 5 and then
never return. Thus lim pi(1n )  0 i . (1 mark)
n 

For C 4  {4,6} : this class is also transient since we can leave the class and go to class 2 and
then never return. Thus lim pi(4n )  0 i and lim pi(6n )  0 i . (1 mark)
n  n 

Question 18 A commodity is being stocked to satisfy a continuous demand. The inventory

policy is described by specifying two non-negative critical values s and S , with s  S . The
inventory level is checked periodically at fixed times t 0 , t1 , t 2 ,... . If the stock on hand at t n is
less than or equal to s , then by immediate procurement the stock level is brought up to level
S . On the other hand if the stock level is greater than s , then no replenishment is done. Let
Z n be the total demand for the commodity during the interval [t n1 , t n ) , and let X n be the stock
on hand just before time t n .
a) Complete the following in terms of S, X n and Z n :
 __________________________________ if s  X n  S , Z n 1  X n

X n 1   __________________________________ if X n  s, Z n 1  S
 __________________________________ otherwise

 X n  Z n 1 if s  X n  S , Z n 1  X n

X n 1  S  Z n 1 if X n  s, Z n 1  S
0
 otherwise
b) Is the parameter space for { X n } discrete or continuous? Explain.
Discrete, since we define X n be the stock on hand just before time t n and t 0 , t1 , t 2 ,... are
period fixed times.

c) What is the state space for the process { X n } ?

{0,1,2,..., S}

d) Define the Markov property for a general process {Yn } .

14
If for any a  S , for any value of n and for any values of t 0  t1  ...  t n  t n1 , it is true
that

PYn1  a | Y0  y0 , Y1  y1 ,...,Yn  y n   PYn1  a | Yn  y n  .

e) In terms of Z n , when will { X n } have the Markov property?

When P[Z n1  k | X 0 ,..., X n ]  P[ Z n1  k | X n ] i.e. if the demand in [t n , t n1 ) only depends
on the stock just before time t n and not times t 0 , t1 ,..., t n 1 .

Question 19 Let X n be the amount of money a gambler has after his n th bet. He plays with
the following strategy:
- If he has R4 or more, he bets R2 which earns him R4, R3 or R0 with probabilities
0.25, 0.3 and 0.45 respectively.
- If he has R1, R2 or R3, he bets R1 which earns him R2 or R0 with probabilities 0.45
and 0.55 respectively.
- He stops playing when he has no more money.
a) Let Yn 1 be his net earnings for the (n  1) th bet. Compute P[Yn1  k | X n  i] for
i  0,1,2,... , k  2,1,0,1,... .
0.25 if X n  4, k  2
0.3 if X n  4, k  1

0.45 if X n  4, k  2

P[Yn 1  k | X n  i ]  0.45 if 0  X n  3, k  1
0.55 if 0  X n  3, k  1

1 if X n  0, k  0

0 otherwise
b) Show that { X n , n  1,2,...} is a Markov chain.
Since X n1  Yn1  X n and the probabilities for Yn 1 depend only on X n the
probabilities for X n 1 only depend on X n and the process is thus Markov.

c) Find the transition probability matrix for { X n , n  1,2,...}, clearly indicating the states.
 1 0 0 0 0 0 0 0 
0.55 0 0.45 0 0 0 0 0 

 0 0.55 0 0.45 0 0 0 0 
 
P 0 0 0.55 0 0.45 0 0 0  where the rows and
 0 0 0.45 0 0 0.3 0.25 0 
 
 0 0 0 0.45 0 0 0.3 0.25 
         

columns refer to the state space S  {0,1,2,3,....} .

d) Classify the states of the process { X n , n  1,2,...} as transient or recurrent and indicate the
equivalence classes. Explain all your reasoning.

15
 State 0 is absorbing and therefore recurrent. It doesn’t communicate with any other
states and therefore forms its own equivalence class as well.

States 1,2,3,… all communicate since pi ,i 1  0 for i  1,2,3... and pi ,i 1  0 for

i  1,2,3 and pi ,i 2  0 for i  4,5,... and therefore form an equivalence class. Since
it is possible to go to state 0 from state 1 this class must be transient since the process
will then never return to this class.

e) What are the limiting probabilities for { X n , n  1,2,...}?

 0  1 (since this state is absorbing) and  i  0, i  1,2,... (since these states are
transient).
The theorem only holds for an irreducible process with recurrent states – we do not
have this for { X n , n  1,2,...}.

f) Explain how you would solve for the limiting distribution of { X n , n  1,2,...} if the state
space was finite and the process was irreducible?
Solve  1  1 and  P    .

Question 20 Consider the following difference equation: xn1  xn   .

a) Classify the equation with respect to its order as well as its homogeneity.
This is a first order non-homogeneous difference equation.

b) If you are given x0 , solve this difference equation recursively by considering the cases   1
and   1 separately.
x n 1  x n  
  (x n 1   )     2 x n 1  
  2 (x n  2   )     3 x n  2     2 

  n 1 x 0   (1    ...   n 1 )
 n 1 1 n
 x0   provided   1
 1
 x  n if   1
 0

Question 21 If { X n , n  01,2,3,...} represents a disease model with state

0.65 0.3 0.05
S  {healthy, sick , dead} and has the transition probability matrix P   0.2 0.3 0.5  ,
 __ __ __ 
a) Fill in a last row for P which makes sense for this disease model.
0.65 0.3 0.05
P   0.2 0.3 0.5 
 0 0 1 

16
 b) What is the probability a healthy person will not die for another 10 transitions?
(10 )
We need g HD from G (10 )  Q 9 R
 1 0 0
P  0.05 0.65 0.3

 0.5 0.2 0.3
so
9
0.65 0.3 0.05 0.0807134 0.0508655  0.05 0.0294684 
G Q R
(10 ) 9
     
 0.2 0.3  0.5  0.0339103 0.0213704   0.5  0.0123807 
(10 )
so g HD  0.0294684 .

c) If the cost incurred per time unit is R1000 when sick, what is the probability that a sick
person will have to spend less than R3000 before they die?
1
 0.65 0.3  3.783784 1.621622 
The sum of the 2nd row of  I     
  is 2.972973,
  0.2 0.3  1.081081 1.891892 
so R1000 x 2.972973 = R2973<R3000, thus we are certain they will spend less than
R3000.

You may make use of the following results for (b) and (c):
9 9
0.3 0.05 0.0009679 0.008303  0.65 0.3 0.0807134 0.0508655 
0.3 0.5   0.0049819 0.0042891
,  0.2 0.3  0.0339103 0.0213704  ,
      
10
0.3 0.05 0.0005395 0.0004636 
0.3 0.5   ,
  0.0027813 0.0023937 
10
0.65 0.3 0.0626368 0.0394737 
 0.2 0.3  ,
  0.0263158 0.0165842 
1
 0.65 0.3  3.783784 1.621622 
I      ,

  0.2 0.3  1.081081 1.891892 
1
 0.3 0.05  1.492537 0.149254 
I      .
 
  0.3 0.5    0.895522 2.089552 
Question 22 Consider a Markov chain { X t : t  0,1,2,...} with state space S  {1,2,3} , initial
 n 1 
n  2 (i ) 0 
 1 
probabilities {0.1,0.5,0.4} , and transition probability matrix P ( n ,n 1)   0 (ii )  .
 n2 
 n 1 
 n  2 n  2 (iii )
a) Complete the transition probability matrix given by providing values for (i), (ii) and (iii).

17
b) Is this Markov chain time-homogeneous? Explain.

c) Prove that for a Markov chain X t : t  T  we have that for any t 0  t1  ...  t n  T and
any values i0 , i1 ,...,in  S it is true that
P[ X t1  i1 , X t2  i2 ,..., X tn  in X t0  i0 ]  pi(0t,0i1,t1 ) pi(1t,1i,2t2 ) pi(2t,2i3,t3 ) ... pi(ntn1,1i,ntn ) .
Theory
d) Calculate the probability the process is in state 3 after two transitions if it starts in state 3.

e) Calculate the probability the process is in state 3 after 1 transition.

n 1
Question 23 Use pi(,nj)   f ij( k ) p (jjn k )  f ij( n ) to get a recursive formula for f ij(n ) and
k 1
hence calculate the probability of a first visit to state 3 after 3 steps if the process does not start
in state 3. You are given the state space S  {1,2,3} and transition probability matrices
 0.1 0.4 0.5 0.24 0.36 0.4  0.216 0.384 0.4 
P  0.2 0.8 0 , P 2  0.18 0.72 0.1 , P 3  0.192 0.648 0.16  .
0.3 0 0.7  0.24 0.12 0.64  0.24 0.192 0.568

18
Question 24 Consider a NCD system with discount levels 0%, 10%, 20% and 40%. If a
policyholder doesn’t claim for two years in a row then they move up to the maximum discount
level but if they only don’t claim one year in a row they move up one discount level or stay in
the maximum discount level. If they claim in any given year, they move down a discount level
(or stay in the lowest discount level). Claims in different years are assumed to be independent,
the maximum number of claims per year by a policyholder is 1 and the probability of a claim
in any year is p .
a) Explain whether or not this process is Markov.

b) Set up a transition probability matrix for this system ensuring the Markov property holds.

19
  p 1 p 0 
c) If the discount levels are instead on 0%, 10% and 20% then P   p  0 1  p  . Take
 0 p 1  p 
p  0.45 .
(i) Show that this process is irreducible. Explain the term irreducible as well.

(ii) Is the process aperiodic? Explain.

(iii) What is the third requirement needed to ensure the existence and uniqueness of the
stationary distribution for this process?

(iv) Find the stationary distribution for this process.

20
Question 25 Every person has two chromosomes, each being a copy of one of the
chromosomes from one of their parents. There are two types of chromosomes labelled X and
Y. A child born with an X and a Y chromosome is male and a child with two X chromosomes
is female. The blood-clotting disorder haemophilia is caused by a defective X chromosome
(X*). A female with the defective chromosome (X*X) will not usually exhibit symptoms of
the disease but may pass the defective gene to her children and so is known as a carrier. A
male with the defective chromosome (X*Y) suffers from the disease and is known as a
haemophiliac. A medical researcher wishes to study the progress of the disease through the
first born child in each generation, starting with a female carrier. You may assume the
following:
• Every parent has an equal chance of passing either of their chromosomes to their children
• The partner of each person in the study does not carry a defective X chromosome
• No new genetic defects occur
The expected progress of the disease through the generations can be modelled as a Markov
chain.
a) Specify a state space with 4 states for the process.

b) Draw a transition diagram indicating the states and the possible transitions between
them.

21
c) Set up the one-step transition probability matrix for the process.

d) Is the process irreducible? Explain and provide the equivalence class(es).

e) Is the process aperiodic? Explain fully.


f) Define a recurrent and transient state in terms of f ii   f ii( n ) . Explain what the
n 0
definition implies in words as well. theory

22
 g) State which states in the given process are recurrent and which are transient.

h) If state i is recurrent and if i  j , prove that state j is also recurrent. theory

i) If state i is transient and if i  j , prove that state j is also transient. theory
j) Determine the limiting probabilities for this process.

Question 26 Consider a no claims discount system with discount levels 0%, 25% and 50%.
The rules for the system are as follows:
- Following a claim-free year the policyholder moves up a discount level or stays in
the highest discount level
- Following a year with one or more claims the policyholder moves down a discount
level or remains in the lowest discount level

Let the probability of a policyholder not claiming in any year be p .

a) Set up a 3 state transition probability matrix for this NCD system.

23
b) If the long-term probability of a policyholder being in the highest discount category is 0.75,
set up equations in which to solve for p which clearly indicate how to find p exactly. You do
not need to show the calculations until the end – simply set up the equations.

c) If an additional discount level is added, of 30%, and a rule is added, namely: two consecutive
claim free years implies a move to the highest discount category, how would you model this
process to ensure the Markov property still holds?

1 0 0 0
 0 0 . 6 0. 2 0 .2 
Question 27 Consider a Markov chain with stochastic matrix P    and
0 0 0 1 
 
0.1 0.1 0.1 0.7 
state space S  {1,2,3,4} .
a) What is the probability of staying out of state 1 for 4 steps if starting in state 4?

24
b) What is the expected number of visits to state 2 if starting in state 4?

You may make use of the following results:

1
0.242 0.09   0.4  0.2
3
0.6 0.2 2.647059 0.588235 
 0.1 0.1  0.045 0.017 ,  0.1 0.9   0.294118 1.176471 
       
1
0.274 0.122 0.538   0.4  0.2  0.2
3
0.6 0.2 0.2 5 2 10
0 0   
1    0.13 0.09 0.61 ,,  0  1  1   2.5 2.5 10
 

 0.1 0.1 0.7 0.139 0.087 0.543   0.1  0.1 0.3  2.5 1.5 10

Question 28 Consider a two-state ( S  {1,2} ) transition probability matrix

1  a a 
P 
(n)
for a Markov chain. Derive a formula for p21 .
 b 1  b

25
Question 29 Consider the following transition probability matrix for a Markov chain with
0 0 1 0 0 0
0.1 0.4 0 0 0.5 0 
 
1 0 0 0 0 0
state space S  {1,2,3,4,5,6} , P   .
0 0 0 0.1 0 0.9
0 0 0 0 1 0
 
 0 0 0 0.5 0 0.5
a) Determine with reasoning the equivalence classes for the process.

b) Define a recurrent and transient state fully.

c) Describe two methods which can be used in a theoretical manner to determine if a state is
recurrent or transient.

d) For the process under consideration, state which classes are recurrent and which are
transient.

e) Complete the statement of the following result: For an aperiodic, irreducible, k-state
Markov chain we have that

26
f) Find lim P n for the process being considered. Note: you do not need to calculate the  ij
n
values, simply provide a formula for these.

Question 30 Consider the following transition probability matrix for a stochastic process
{ X t : t  0,1,2,3,...} with state space S  {1,2,3} :
 1 
1  m  n x 0 
 2 
P ( n, m)   0 y 
 2mn 
 z 0
1 
 1  m  n 
a) Provide formulas for the unknowns x, y and z above.

27
b) Explain why this stochastic process has the Markov property.

c) Is this stochastic process time-homogeneous? Explain.

d) Calculate PX n  2  2 | X n  1.

e) If you were given P ( n , n 1) instead of P ( n , m ) how would you calculate PX n  2  2 | X n  1?
State the result you make use of.

f) Provide a formula to calculate P[ X 3  3] assuming again that you have P ( n , m ) for the process.

Question 31 Consider a finite time-homogeneous stochastic process { X t : t  0,1,2,3,...}. If

there exists an N such that for all n  N it is true that P n has no zero elements, prove that the
process is irreducible and aperiodic.

28
Question 32 Consider a stochastic process with S  {1,2,3,4,5,6} with transition probability
matrix
 0. 2 0 0 0 0 0.8
0.4 0.1 0.1 0.1 0.3 0 
 
 0 0.5 0.5 0 0 0
P .
0 0 0 0 1 0
0 0 0 0 1 0
 
0.6 0 0 0 0 0.4
a) Give the equivalence classes for this Markov chain, with explanation.

 P 
b) Put the given transition probability matrix in the form  1  . Clearly indicate the new
 R Q
order of the states.

c) Define the probability gij(n) .

d) Calculate the probability the process, if starting in state 2, will only visit state 5 after 3
transitions.

29
e) Define the probability hij(n ) .

f) How many times is it expected the process will visit state 3, if the process starts in state 2.

Hint:
1
 0.9  0.1  0.1 1.25 0.25 0.125
 0.5 0.5 0   1.25 2.25 0.125 ,


 0 0 1   0 0 1 
1
 0.9 0.1 0.1 0.25 2.25 1.125 
 0.5  0.5 0   1.25 2.25 3.125 ,
   
 0 0  1  0 3 1 
1
0.6 0  0.3 2.25 0.25 1.125
0 1 0   0.25 1.25 1.125


 0 0 0   1 3 0 

g) Find lim pij( n ) for j  1,2,3,4,5,6 and i  C j i.e. only for i in the same equivalence class as
n 

j.

30
Question 33 Suppose that players A and B gamble and that at each play the players bet an
amount out of their current total money in hand. Player A will bet R1 if his previous bet was
lost but R2 if his previous bet was won, however Player A can only bet an amount if Player B
has the equivalent amount to bet, else the bet stays at R1. Set up a transition probability matrix
for Player A to model this game as a Markov chain given that there is a total amount of R4
between the players and the probability of winning a bet for player A is p .

Question 34 a) Define time-homogeneity.

b) Explain how the transition probability matrix P ( m , n ) changes when the process is time-
homogeneous. Why is this useful?

c) State the time-homogenous Chapman-Kolmogorov equations in both element-wise format

and matrix format. Explain why these equations are useful.

31
Question 35 Describe two ways in which one can determine whether a state i is recurrent or
transient. Prove both methods.

Question 36 Consider the following transition probability matrix for a Markov chain
{ X t : t  0,1,2,...} with stats space S  {1,2,3,4,5} :
1 0 0 0 0
0 1 0 0 0 

P  0.4 0.3 0.3 0 0 .
 
 0 0.2 0 0.6 0.2
 0 0.1 0 0.1 0.8

a) Find, with reasoning the equivalence classes for this process.

b) Prove that recurrence is a class property.

32
c) Classify your classes in (a) as recurrent or transient, with explanation.

d) Find the lim P n for this process.

n

33
 Hint:

1 1
0.7 0 0  1.4286 0 0  0.3 0 0 3.3333 0 0 
0  
0.4  0.2   0    
3.3333 3.3333 ,  0 0.6 0.2   0 1.7391  0.4348 

 0  0.1 0.2   0 1.6667 6.6667   0 0.1 0.8  0  0.2174 1.3043 

Question 37 Consider the following transition probability matrix for a Markov chain with state
space S  {1,2,3,4}
 0. 6 0 0 0.4
 0.1 0.1 0 0.8
P .
0 0 0.1 0.9
 
 0. 2 0 0 0.8
a) Determine with explanation the equivalence classes for this process.

b) For { X n : n  0,1,2,...} a time-homogeneous Markov chain, prove that if the state space of
the Markov chain consists of a FINITE number of states, not all states can be transient.

34
c) Classify the states of the process as recurrent or transient, using the result in (b).

d) Prove that for { X t : t  0,1,2,...} a time-homogeneous Markov chain with C1 and C 2 two
equivalence classes of states of the chain, if i  C1 and j  C 2 and pi(,mj)  0 for some m , then
p (jn,k)  0 for all k  C1 and for all n . PROOF
e) Determine, with explanation, the period of each state in the process.

Question 38 a) Define a recurrent and transient state, in terms of formulas and a description.

b) Describe two methods to classify as state as recurrent or transient.

c) Define positive and null-recurrence.

35
Question 39 Consider the following no claims discount system.
- The probability of a claim in any given year is p .
- If a policyholder did claim the previous year they move down a single category or stay in the
lowest category. - If a policyholder doesn’t claim two years in a row they move to the highest
category.
- If a policyholder did claim two years ago and but didn’t claim the previous year they move
up a single category or stay in the highest category.

(a) Set up a Markov transition probability matrix if the four discount levels are 0%, 25%, 30%
and 40%.

(b) If p  0.2 determine the stationary distribution for this no claims discount system.

Question 40 a) What is meant by a recursive formula?

36
 f ij(n)
b) Derive a recursive formula for .

c) Explain how you would use (b) to determine if a state is recurrent or transient.

d) State and prove a second method to determine if a state is recurrent or transient.

P 0 
P 1 
Question 41 a) Explain the representation  R Q of a transition probability matrix.

n
P 0 
P  1
n

b) Derive a general formula for  R Q  i.e. a recursive formula for each sub-matrix.

37
 lim P n
c) Use (b) to give a description of n  .

 0 . 1 0 . 2 0 . 3 0. 4 
1 0 0 0 
P 
0 0 0 . 4 0. 6 
 
Question 42 Consider the transition probability matrix 0 0 0. 5 0 .5 
with
n
lim P
S={1,2,3,4}. Determine, with full reasoning at every step, n . You do not need to calculate
 ij
numerical values for the ’s.

38
Question 43 a) Provide a formula for pij( 4) in terms of only 1-step transition probabilities pij .
State which result you use.

b) Explain why for a time-homogeneous Markov chain only the 1-step transition probability
matrix is needed to completely specify the distribution of the process.

c) Consider the following transition probability matrix of a Markov chain with S  {1,2,3} :
 0 0.1 0.9
P  0.3 0.2 0.5 . Find the period of state 1.
 0 0.5 0.5

39
Question 44 Suppose that players A and B gamble and that at each play the bet is for R1. Both
of them put up R1 at each play and the one that wins gets both rands so that after one play the
winner will have R1 more than before the play and the loser will have R1 less than before the
play. Let the total amount the two players together have available to play by R4. Let Ym be the
amount player A has after m bets. They continue to play until one of them has won all the
money available. The following transition probability matrix governs this Markov chain (with
probability of winning/losing changing with the amount A has to play with):
1 0 0 0 0
 0. 5 0 0. 5 0 0 

P   0 0.6 0 0.4 0 
 
0 0 0.7 0 0.3
 0 0 0 0 1 

(i) What is the state space for this process?

(ii) What is the probability player A will have won everything for the first time after 3
bets if starting with R2? Comment on the answer/explain what you observe as well.

(iii) Let  i be the probability that if player A starts with R i he will eventually win all
the money. Set up a difference equation for  i by conditioning on the outcome of
the previous bet, if the probability of losing a bet is now a constant p .

40
 (iv) Solve the difference equation obtained in (iii) for p  0.5 .

Question 45 Suppose the number of claims by a policyholder in a single year follows a Poisson
distribution with parameters 0.5. If a policyholder claims once in the year they move down a
single category or stay in the lowest category. If a policyholder claims twice or more in the
year they move to the lowest category. If a policyholder doesn’t claim in the year they move
up a single category or stay in the highest category.
a) Can this process be considered as a Markov process? Motivate your answer.

b) Set up the transition probability matrix if the three discount levels are 0%, 25% and 40%.

 0 0.6 0.4
Question 46 Consider a transition probability matrix P   0 1 0  with S  {1,2,3}
0.2 0 0.8
for a Markov chain.
a) Explain why this Markov chain is time-homogeneous.

b) Determine with reasoning the equivalence classes and classify them as either transient or
recurrent, with reasoning.

41
c) Determine the probability this process starts in state 3 and visits state 2 for the first time at
time point 3.

d) Calculate the expected amount of time spent in state 1.

1 1
 1  0.4 1.6 3.3   0.6 0 1.667 0 
Given:     ,    
1.6 8.3  0.8 0.8
 0.2 0.2    1.667 1.25

42
Question 47 Consider the following transition probability matrix for a Markov chain with
states S  {1,2,3}
 0 .2 0 0 .8 
P  0.7 0.3 0  .
0.4 0.4 0.2
a) Show that the process is irreducible.

 
b) What two additional properties are needed in order for lim P      to exist?
n
n 
 

 
c) Assuming the additional properties mentioned in (b) hold, find lim P      .
n
n 
 
d) Use your answer in (c) to determine if the states of the process are null- or positive-recurrent.
Motivate your answer.

43
Question 48 Suppose the number of claims by a policyholder in a single year follows a
binomial distribution with parameters 3 and 0.5. If a policyholder claims once in the year they
move down a single category or stay in the lowest category. If a policyholder claims twice or
more in the year they move to the lowest category. If a policyholder doesn’t claim in the year
they move up a single category or stay in the highest category. The insurance decides on three
discount levels 0%, 25% and 40%.
a) Can this process be considered as a Markov process? Motivate your answer.

b) Comment on the use of the binomial distribution to model the claims numbers and the
implication for modeling an NCD system.

c) Set up the transition probability matrix for the three discount levels are 0%, 25% and 40%.

44
45