1

Final
Elementary Stochastic Process (MTH-412A)

Name:
Roll Number:
Time 120 minutes
Maximum Marks: 40

Important Instruction: This question paper has 7 questions, and each question has 6
marks. Questions 1-3 are short answer types and they do not have any negative marks,
Question 4-7 are multiple choice where more than one answers might be correct. Each
correct answer will give you one mark and each wrong answer will give you -1 mark. If you
can identify all the correct answers without any wrong answer, then you will get 6 points of
that question. We have used the same notations as we have used in the class. You need to
write your name and roll number in each page, if you do not write in any page 2 marks will
be deducted. Do not do any rough work in these pages. If you do any rough work in these
pages two marks will be deducted. Write your answers in this question paper, and return
this question paper along with the booklet.

Question 1: Suppose {Xn ; n ≥ 1} is a Markov Chain with the state space S = {1, 2, 3, 4, 5, 6},
and having the transition probability matrix
 1 1 1 
2
0 4
0 4
0
 1 3 1 3 1 1 
 10 10 10 10 10 10 
 1 1 1
0 0 0 


P=  4 2 4
1 .

 3 1 3 1 1
 10 10 10 10 10 10 
 1 1 1
0 0 0 

 4 4 2
3 1 1 1 1 3
10 10 10 10 10 10

Then (correct up to two decimal places or in fraction):

(n) (n) (n)

1. limn→∞ p11 = [ ], 2. limn→∞ p31 = [ ], 3. limn→∞ p24 = [ ],
(n) (n) (n)
4. limn→∞ p25 = [ ], 5. limn→∞ p65 = [ ], 6. limn→∞ p66 = [ ].

Question 2: Consider a sequence of tosses of a coin independently with the probability of

head 1/3 and the probability of tail 2/3. Suppose Xn and Yn denote the number of heads
and number of tails, respectively, after n tosses. If Zn = |Xn − Yn |, then (up to two decimal
points or in fraction)

P (Z101 = 1|Z100 = 2) = [ ]
 2

Question 3: Let {Xn } and {Yn } be two independent Markov chains with the same state
space S = {0, 1} and having the same transition probability matrix
" #
1 1
P= 2 2 .
1 1
2 2

Suppose Zn = max{Xn , Yn }. Then (correct up to two decimal places or in fraction).

limn→∞ P (Zn+1 = 1|Zn = 1) = [ ]

Question 4: Consider a Markov Chain {Xn ; n ≥ 1} with the following transition probability
matrix: P (Xn+1 = i + 1|Xn = i) = P (Xn+1 = i − 1|Xn = i) = P (Xn+1 = i|Xn = i) = 13 , for
i = 0, ∓1, ∓2, . . .. Then which of the following statement(s) is (are) correct for n ≥ 2.

1. P (X3n+1 = 0|Xn = 0) = 0.
   2n
2n 1
2. P (X3n = 0|Xn = 0) > n 3
.
   2n
2n 1
3. P (X3n = 0|Xn = 0) ≤ n 3
.
    2n
2n 1
4. P (X3n = 0|Xn = 0) < n
+1 3
.

5. All the states communicate with each other.

Then which is of the above statements are correct: [ ]

Question 5: Let us consider two Markov chains {Xn } and {Yn }, with the same states space
S = {0, 1, 2, . . .}. The transition probability matrices are as follows:
   
p00 p01 p02 . . . 1 0 0 ...

 p10 p11 p12 . . . 


 p10 p11 p12 . . . 

P= 
 p20 p21 p22 . . . 
 and P
e = 
 p20 p21 p22 . . . 

.. .. .. .. .. ..
   
. . . . . .

Consider the following statements:

1. If {Xn } is irreducible and aperiodic, then {Yn } is also aperiodic.

2. If {Xn } is irreducible and transient, then the states {1, 2, 3, . . .} in {Yn } are also tran-
sient.

3. If the states {1, 2, 3, . . .} in {Yn } are recurrent, then they are recurrent in {Xn } also.

4. If all the states in {Xn } are recurrent, then the states {1, 2, 3, . . .} in {Yn } are also
recurrent.

Then which is of the above statements are correct: [ ]

 3

Name:
Roll Number:

Question 6: A M × M matrix P = ((pij )), is called a stochastic matrix if pij ≥ 0, and

M
X M
X
pij = 1, for i = 1, . . . , M . A stochastic matrix P is called a doubly stochastic if pij = 1,
j=1 i=1
for j = 1, . . . , M . Consider the following statements.

1. If P is a stochastic matrix then Pm is also a stochastic matrix for m = 1, 2, . . . .

2. If P is a doubly stochastic matrix then Pm is also a doubly stochastic matrix for

m = 1, 2, . . . .

3. If P is a doubly stochastic matrix then for any m = 1, 2, . . . , then there exists a doubly
stochastic matrix Qm , such that P = (Qm )m .
(n)
4. If P is a doubly stochastic matrix, and if pij > 0, for 1 ≤ i, j ≤ M , then limn→∞ pij =
0.
(n)
5. If P is a doubly stochastic matrix, and if pij > 0, for 1 ≤ i, j ≤ M , then limn→∞ pij =
1
M
.
(n) 1
6. If P is a stochastic matrix, and if pij > 0, for 1 ≤ i, j ≤ M , then limn→∞ pij = M
.

Then which is of the above statements are correct: [ ]

Question 7: Consider a Markov Chain {Xn } having state space S = {0, 1, 2, 3, . . .}, with
the following transition probability matrix

p 1−p 0 0 0 0 ...
 
p
 0 1−p 0 0 0 ...
P = p 0 0 1−p 0 0 ...
 
.

p 0 0 0 1 − p 0 ...
.. .. .. .. .. ..
 
. . . . . .

Here 0 < p < 1. Define a random variable N = min{n : n ≥ 1, Xn = 0|X0 = 0}. E(N ) and
V (N ) denote the mean and variance of the random variable N .
 4

Name:
Roll Number:

Consider the following statements.

1. {Xn } is an irreducible Markov Chain for all 0 < p < 1.

2. {Xn } is an irreducible Markov Chain only for 0.5 ≤ p < 1, and not for 0 < p < 0.5.

3. E(N ) exists and it is p1 .

1−p
4. E(N ) exists and it is p
.
1−p
5. V (N ) exists and it is p2
.
(1−p)2
6. V (N ) exists and it is p
.

Then which is of the above statements are correct: [ ].

All the best.

 5

Solutions:

Question 1: Suppose {Xn ; n ≥ 1} is a Markov Chain with the state space S = {1, 2, 3, 4, 5, 6},
and having the transition probability matrix
 1 1 1 
2
0 4
0 4
0
 1 3 1 3 1 1 
 10 10 10 10 10 10 
 1 1 1
0 0 0 


P=  4 2 4
1 .

 3 1 3 1 1
 10 10 10 10 10 10 
 1 1 1
0 0 0 

 4 4 2
3 1 1 1 1 3
10 10 10 10 10 10

Provide the answers up to two decimal places:

(n) 1
1. limn→∞ p11 = 3

(n) 1
2. limn→∞ p31 = 3

(n)
3. limn→∞ p24 = 0
(n) 1
4. limn→∞ p25 = 3

(n) 1
5. limn→∞ p65 = 3

(n)
6. limn→∞ p66 = 0.

Question 2:

P (Z101 = 1||Z100 = 2) = P (Z101 = 1, X100 − Y100 = 2|Z100 = 2) +

P (Z101 = 1, Y100 − X100 = 2|Z100 = 2)
= P (Z101 = 1|X100 − Y100 = 2)P (X100 − Y100 = 2|Z100 = 2) +
P (Z101 = 1|Y100 − X100 = 2)P (Y100 − X100 = 2|Z100 = 2)
2 1 1 4 2
× + × = = 0.40
3 5 3 5 5

Question 3: Let {Xn } and {Yn } be two Markov chains with the same state space S = {0, 1}
and having the same transition probability matrix
" #
1 1
P= 2 2
1 1
2 2

and they are independently distributed. Suppose Zn = max{Xn , Yn }. Then limn→∞ P (Zn+1 =
1|Zn = 1) = 43 = 0.75 (up to two decimal places).
 6

Observe that Pn = P and what ever be P (X1 = 0) and P (Y1 = 0), P (Xn = 0) = P (Xn =
1) = P (Yn = 0) = P (Yn = 1) = 12 . Therefore,

P (Zn+1 = 0|Zn = 1) = P (Zn+1 = 0, Xn = 0, Yn = 1|Zn = 1) +

P (Zn+1 = 0, Xn = 1, Yn = 0|Zn = 1) +
1 1
P (Zn+1 = 0, Xn = 1, Yn = 1|Zn = 1) = 3 × × .
4 3

Question 4: Consider a Markov Chain {Xn ; n ≥ 1} with the following transition probability
matrix: P (Xn+1 = i + 1|Xn = i) = P (Xn+1 = i − 1|Xn = i) = P (Xn+1 = i|Xn = i) = 13 , for
i = 0, ∓1, ∓2, . . .. Then which is of the following statement(s) is (are) correct.

1. P (X3n+1 = 0|Xn = 0) = 0. (F)

   2n
2n 1
2. P (X3n = 0|Xn = 0) > n 3
. (T)
   2n
2n 1
3. P (X3n = 0|Xn = 0) ≤ n 3
. (F)
    2n
2n 1
4. P (X3n = 0|Xn = 0) ≤ n
+1 3
. (F)

5. All the states communicate with each other. (T)

Then which is of the above statements are correct: [ 2, 5]

It follows by counting the paths the way it can come back to the same state after 2n or
2n + 1 steps.

Question 5:

1. False. Take  1 1 1 1 
2 22 23 24
...
 1 1
 2 0 2
0 ... 

P =  1
 0 0 1
...


 2 2
.. .. .. .. ..

. . . . .
P∞ P∞
2. True. If i, j ∈ {1, 2, . . .}, then penij ≤ pnij , for all n. Hence, n=1 penij ≤ n=1 pnij < ∞.

3. True. Follows from the same inequality as the previous one.

n
4. False. Same example as in (a). Note that f00 = 2−n . Hence all the states are recurrent
in A.

Then which is of the above statements are correct: [ 2, 3]

 7

Question 6: A M × M matrix P = ((pij )), is called a stochastic matrix if pij ≥ 0, and

M
X M
X
pij = 1, for i = 1, . . . , M . A stochastic matrix P is called a doubly stochastic if pij = 1,
j=1 i=1
(n)
for j = 1, . . . , M . Then which of the following statements are always correct. Here pij
denotes the (i, j)th element of the matrix Pn .

1. If P is a stochastic matrix then Pm is also a stochastic matrix for m = 1, 2, . . . . (T)

2. If P is a doubly stochastic matrix then Pm is also a doubly stochastic matrix for

m = 1, 2, . . . . (T)

3. If P is a doubly stochastic matrix then for any m = 1, 2, . . . , there exists a doubly

stochastic matrix Qm , such that P = (Qm )m . (F)
(n)
4. If P is a doubly stochastic matrix, and if pij > 0, for 1 ≤ i, j ≤ M , then limn→∞ pij =
0. (F)
(n)
5. If P is a doubly stochastic matrix, and if pij > 0, for 1 ≤ i, j ≤ M , then limn→∞ pij =
1
M
. (T)
(n) 1
6. If P is a stochastic matrix, and if pij > 0, for 1 ≤ i, j ≤ M , then limn→∞ pij = M
.
(F)

Then which is of the above statements are correct: [ 1, 2, 5]

Question 7: Consider a Markov Chain {Xn } having state space S = {0, 1, 2, 3, . . .}, with
the following transition probability matrix
p 1−p 0 0 0 0 ...
 
p
 0 1−p 0 0 0 ...
p 0 0 1 − p 0 0 ...
 
P=

.
p 0 0 0 1−p 0 ...
.. .. .. .. .. ..
 
. . . . . .

Here 0 < p < 1. Define a random variable N = min{n : n ≥ 1, Xn = 0|X0 = 0}. Consider
the following statements.

1. {Xn } is an irreducible Markov Chain for all 0 < p < 1. [T]

2. {Xn } is an irreducible Markov Chain only for 0.5 ≤ p < 1, and not for 0 < p < 0.5.
[F]

3. E(N ) exists and it is p1 . [T]

 8

1−p
4. E(N ) exists and it is p
. [F]
1−p
5. V (N ) exists and it is p2
. [T]
(1−p)2
6. V (N ) exists and it is p
. [F]

Then which is of the above statements are correct: [ 1, 3, 5]

In this case P (N = k) = p(1 − p)k−1 , for k = 1, 2, . . . . It is a geometric random variable.

E(N ) = p1 and V (N ) = 1−p
p2
.