AMS 342 Joe Mitchell

Operations Research II: Stochastic Models: Practice Midterm

1. (25 points) Assume that the number of telephone calls arriving at a switchboard by time t is
described by a Poisson process, {N (t), t ≥ 0}. We observe that, on average, the time between
arrivals is five minutes.
(a). (5 points) What is λ, the rate of the process? (Give UNITS!)
Since there is one call per 5 minutes, the rate is λ = 15 min−1 = 0.2 min−1 . (or λ = 12 hour−1 )
(b). (20 points; 10 points per question) DO ANY 2 (TWO) of the following problems. Indicate
which two problems you want graded.
(i). What is the probability that the time between the arrival of the 5th call and the arrival of the
6th call is more than 20 minutes?
T6 is the time (in minutes) between the arrival of the 5th call and the arrival of the 6th call.
We know that each of the interarrival times, Ti , is exp(λ), with λ = 0.2min−1 . Thus, we compute

P (T6 > 20) = e−λ·20 = e−4

0
(Another way to do it: P (T6 > 20) = P (T1 > 20) = P (N (20) = 0) = e−λ·20 (λ·20)
0! .)
(ii). What is the probability that at least one call arrives in the first ten minutes (0 < t ≤ 10
minutes) and that no calls arrive in the following twenty minutes (i.e., in the interval 10 < t ≤ 30
minutes)?
P (N (1) ≥ 1 and N (30) − N (10) = 0) = [1 − P (N (10) = 0)]P (N (20) = 0) = [1 − e−2 ] · e−4
(iii). What is the probability that the first call occurs after time t = 1 hour?
Be careful of units! We want P (T1 > 60), since we are using minutes as the time unit. This is
P (T1 > 60) = e−λ·60 = e−12 .
2. (25 points) Consider an inventory model in which the replenishment of stock takes place at
the end of periods labeled n = 0, 1, 2, . . ., and we assume that the total demand for the single
commodity during period n is a random variable Dn , independent of the time period. Assume
that P {Dn = 0} = .1, P {Dn = 1} = .2, P {Dn = 2} = .7. Assume that an (S, s) policy is
used: If the end-of-period stock is less than or equal to s, then an amount sufficient to increase
the quantity of stock on hand up to the level S is immediately ordered; otherwise, no new stock is
ordered. Assume that S = 2, s = −1. Let Xn denote the quantity on hand at the end of period
n, just before restocking. A negative value of Xn is interpreted as an unfilled demand that will be
satisfied immediately upon restocking. This is the inventory example we studied in class; recall that
{Xn , n ≥ 0} is a Markov chain. Assume that X0 = 1.
(a). (12 points) Write the transition matrix P , labeling the rows and columns with the corre-
sponding states.
The rows and columns are labelled -2, -1, 0, 1, 2:
 
0 0 .7 .2 .1
 0 0 .7 .2 .1 
 
P =  .7 .2 .1 0 0 
 
 0 .7 .2 .1 0 
 

0 0 .7 .2 .1
Why? If we have -2 items on hand at the end of today, we order enough to start tomorrow
with 2 items; thus, we end tomorrow with 2 items if nothing is demanded (which happens with
probability P (Dn = 0) = .1), we end tomorrow with 1 item if 1 item is demanded (probability .2),
and we end tomorrow with 0 items if 2 items are demanded (probability .7). Apply this reasoning
to get each row of the matrix above.
(b). (8 points) Give a system of equations whose solution would allow you to compute the long-
run fraction of time that the chain is in state i, for every i. Be very explicit! (You need not solve
equations.)
The long-run fraction of time in stae i is πi , where the πi ’s are given by solving the following
system of equations (π = πP , i πi = 1):
P

π−2 = (.7)π0
π−1 = (.2)π0 + (.7)π1
π0 = (.7)π−2 + (.7)π−1 + (.1)π0 + (.2)π1 + (.7)π2
π1 = (.2)π−2 + (.2)π−1 + (.1)π1 + (.2)π2
π2 = (.1)π−2 + (.1)π−1 + (.1)π2
π−2 + π−1 + π0 + π1 + π2 = 1

(c). (5 points) What is the average size of a restocking order that is placed? (Compute the
average over all orders, including those “orders” of size 0.) You may leave your answer in terms
of the πi ’s.
How big is the restocking order? If we are in state -2 (at the end of the day, we have unfilled
demand of 2), we order 4 items; if we are in state -1, we order 3 items; otherwise, we do not order
any new stock (the size of the restocking order is 0).
On average, we are in state -2 π−2 fraction of the time, etc. Thus, the average restocking order
is 4 · π−2 + 3 · π−1 .
(Note: There is a matter of interpretation here: is an order for 0 items really an “order”? If we
want to compute the expected size of non-zero orders, meaning that we condition on being in state
-2 or -1, we get an average size of order (4π−2 + 3π−1 )/(π−2 + π−1 ).)
3. (20 points) Consider the Markov chain on state space S = {0, 1, 2, 3} whose transition diagram
is shown below. Assume that the chain starts in state 0 (X0 = 0).

1/4 1
1/2 1/2
2 1 0 3
1/2 1/4
1

(a). (12 points) What is the probability that the chain is absorbed into state 3?
First, we note that there are three classes: {0} (transient), {1, 2} (positive recurrent), and {3}
(positive recurrent).
We want to compute p0 = P ( absorbed into state 3 |X0 = 0). We condition on the first
transition, which is either to the left (which means for sure we do not get absorbed into state 3),
right (which means for sure we do get absorbed into state 3), or a return to state 0 (which means
we “start over”, having a probability of p0 of being absorbed into state 3):

p0 = (1/2) · 0 + (1/4) · 1 + (1/4) · p0 ,

2
which gives p0 = 1/3.
(An alternative solution is based on conditioning on the move that takes the chain into an
absorbing class: P ( absorbed to state 3 | absorbed to state 3 or to class {1, 2}) = (1/4)/((1/4) +
(1/2)) = 1/3.)
(b). (8 points) What is the probability that the number of steps until absorption (into any
absorbing class) is even?
Let p = P ( the number of steps until absorption is even |X0 = 0). Condition on the first move:

p = (1/4)(1 − p) + (1/4) · 0 + (1/2) · 0,

which gives p = 1/5.

4. (20 points; 10 points per question) Consider an ergodic Markov chain on state space S =
{0, 1, 2, 3}. Let the vector π = (π0 , π1 , π2 , π3 ) be the long-run distribution (i.e., the stationary
distribution). Let Pijn be, as usual, the n-step transition probability from state i to state j. Assume
that the distribution of where the chain starts at time 0 is given by the vector α(0) = π (i.e., start
the chain according to the stationary distribution).
DO ANY 2 (TWO) of the following problems. Indicate which two problems you want graded.
You may express your answers in terms of the πi ’s and the Pijn ’s.
(i).
P (X2 = 1, X3 = 2) π1 P12
P {X2 = 1 | X3 = 2} = =
P (X3 = 2) π2
(ii).
2 2 2 2 2 2
X
P {X5 > X7 } = P (X5 > X7 |X5 = i)P (X5 = i) = 0π0 +P10 π1 +(P20 +P21 )π2 +(P30 +P31 +P32 )π3
i

(Here we have used the fact that X5 has distribution π in this case, since α(0) = π, so α(n) = π,
for all n ≥ 0.)
(iii).
2 2
P {X5 = 0, X7 = 2 | X3 = 2} = P20 P02
(iv).
V ar[X3 ] = (02 π0 + 12 π1 + 22 π2 + 32 π3 ) − (0π0 + 1π1 + 2π2 + 3π3 )2
(since X3 has distribution π in this case, since α(0) = π, so α(n) = π, for all n ≥ 0.)
5. (10 points) Consider a discrete-time Markov chain on state space S = {0, 1, 2} whose single-step
probability transition matrix is given below.

1/2 0 1/2
 

P = 1/3 1/3
 1/3 
1/2 0 1/2
(a). (5 points) List the communicating classes, and give the period of each. Also state whether
each class is transient, positive recurrent, or null recurrent.
The classes are {1} (transient, period=1 (aperiodic)), and {0, 2} (positive recurrent, period=1).
(b). (5 points) Compute
2
P {X5 = 0 | X3 = 0} = P00 = (1/2)(1/2) + 0(1/3) + (1/2)(1/2) = 1/2

3
AMS 342 Joe Mitchell
Operations Research II: Stochastic Models: Another Practice Midterm
1. (20 points) Consider two lightbulbs: one made by Westinghouse, and one made by General
Electric. Let X denote the lifetime of the Westinghouse lightbulb, and let Y denote the lifetime
of the General Electric bulb. Assume that X has an exponential distribution with mean 2 hours,
and that Y has an exponential distribution with mean 30 minutes. Also assume that X and Y are
independent. (Warning: Be careful about UNITS!)
(a). (8 points) Find P {X 2 < Y 2 }.
We are told that X (measured in hours) is exp(1/2) and Y (measured in hours) is exp(2), and
they are independent.
Since X and Y are nonnegative, the event {X 2 < Y 2 } is equivalent to the event that {X < Y }
1/2
Thus, we want to compute P (X 2 < Y 2 ) = P (X < Y ) = (1/2)+2 = 0.2.
(b). (12 points) Assume that at time 0, the two bulbs are initially installed. Then, at time 1
hour, the bulbs are inspected. If both are burned out, then the Westinghouse bulb is replaced (but
the General Electric bulb is left alone). Then, at time 2 hours, we inspect the two bulbs again.
What is the probability that we find both bulbs burned out at time 2?
We need to compute a probability about what things are like at time 2... it sure would be useful
to know what exactly happened at time 1...
Thus, we condition! There are 4 cases, which define 4 events corresponding to what happened
at time 1: Let us make some precise definitions:

F1 = {both bulbs are burned out at time 1 hour} = {Westinghouse replaced at time 1}

F2 = {Westinghouse burned out at time 1 hour, but GE bulb still good at time 1}
F3 = {GE bulb burned out at time 1 hour, but Westinghouse still good at time 1}
F4 = {both bulbs still good at time 1}
Note that

P (F1 ) = P (X < 1, Y < 1) = P (X < 1) · P (Y < 1) = (1 − e−1/2 )(1 − e−2 ),

P (F2 ) = P (X < 1, Y > 1) = P (X < 1) · P (Y > 1) = (1 − e−1/2 )e−2 ,

P (F3 ) = P (X > 1, Y < 1) = P (X > 1) · P (Y < 1) = e−1/2 (1 − e−2 ),
P (F4 ) = P (X > 1, Y > 1) = P (X > 1) · P (Y > 1) = e−1/2 e−2 ,
where we have used the independence of X and Y . Note that P (F1 ) + P (F2 ) + P (F3 ) + P (F4 ) = 1,
as it should, since we claim these 4 cases form a partition of the sample space.
Let E = { both bulbs are burned out at time 2 hours }. We want to compute

P (E) = P (E|F1 )P (F1 ) + P (E|F2 )P (F2 ) + P (E|F3 )P (F3 ) + P (E|F4 )P (F4 )

Now, P (E|F1 ) = P (X < 1) = (1 − e−1/2 ), since, given F1 , both are burned out at time 2 if
and only if the (new) Westinghouse bulb burns out within 1 hour. Similarly, P (E|F2 ) = P (Y <
1) = (1 − e−2 ) (by memoryless property, since the GE bulb is like a “new” bulb installed at time

4
1), P (E|F3 ) = P (X < 1) = (1 − e−1/2 ) (by memoryless property, since the Westinghouse bulb is
like a “new” bulb installed at time 1), and P (E|F4 ) = P (X < 1, Y < 1) = P (X < 1)P (Y < 1) =
(1 − e−1/2 )(1 − e−2 ) (again, by memoryless property).
The final answer is 4i=1 P (E|Fi )P (Fi ), where each term is as given above.
P

2. (20 points) Consider the maze shown below. There are three cells (Cell 1, Cell 2, and Shock)
and two deadly (quite permanent) outcomes (Death By Poison, and the dreaded Death By Torture).
A rat is initially placed in cell 1. Each minute, the rat moves from his current cell into one of
the adjoining cells or through a door that leads to one of the two types of death, according to the
following probabilities. From Cell 1 he goes next to Cell 2, gets shocked, or experiences Death By
Poison, each with probability 1/3. From Cell 2 he goes next to Cell 1, gets shocked, or experiences
Death By Torture, each with probability 1/3. From the Shock cell he goes next to Cell 2 (with
certainty). (That is, the door between Cell 1 and the Shock cell is one-way in the direction from
Cell 1 to the Shock cell.)

Death by Poison Cell 1 Cell 2 Death by Torture

Shock Treatment

(a). (8 points) Model the status of the rat as a Markov chain; that is, state precisely what
the state space is (and what each state means) and what the transition probability matrix, P , is.
Classify the states as transient or recurrent.
Let S = {0, 1, 2, S, T }, where
0 if the rat is in Death by Poison


1

 if the rat is in Cell 1

Xn = 2 if the rat is in Cell 2
S if the rat is in Shock




T if the rat is in Death by Torture
The transition matrix, with rows and columns labelled in order 0,1,2,S,T is given by:
 
1 0 0 0 0
 1/3 0 1/3 1/3 0 
 
P =  0 1/3 0 1/3 1/3 
 
 0 0 1 0 0 
 

0 0 0 0 1
Thus, there are three classes: {0} (positive recurrent), {1, 2, S} (transient), and {T } (positive
recurrent).
(b). (12 points) Find the probability that the rat gets shocked before he dies.
Let pi = P ( the rat gets shocked before he dies |X0 = i). We want p1 . Trivially, p0 = 0, pS = 1,
and pT = 0.
We condition on the first step:
p1 = 0 · (1/3) + 1 · (1/3) + p2 · (1/3)
p2 = p1 · (1/3) + 1 · (1/3) + 0 · (1/3)

5
Solving, we get p1 = 1/2, p2 = 1/2, so the answer is p1 = 1/2.
3. (20 points) Consider a Markov chain on the state space S = {0, 1, 2, 3} for which the one-step
transition matrix is as follows:
0 1 0 0
 
0 0 1 0 
P = 0 1 0 1 

2 2
0 12 12 0
Let Xn be the state of the chain at time n. Determine the following quantities:
(a). (8 points) The expected number of steps it takes until the first return to state 2, given that
the chain starts in state 2;
First, we identify the classes: {0} (transient), {1, 2, 3} (positive recurrent, aperiodic).
We want to compute m2 = 1/π2 . Thus, we start by computing the π vector (which will end up
showing that π0 = 0, since state 0 is transient). The πi ’s are given by solving the following system
of equations (π = πP , i πi = 1):
P

π0 = 0
π1 = 1π0 + (1/2)π2 + (1/2)π3
π2 = π1 + (1/2)π3
π3 = (1/2)π2
1 = π0 + π1 + π2 + π3

We get π0 = 0, π1 = 1/3, π2 = 4/9, π3 = 2/9. So m2 = 1/π2 = 9/4.

(b). (4 points) limn→∞ V ar(Xn );

lim V ar(Xn ) = (02 π0 + 12 π1 + 22 π2 + 32 π3 ) − (0π0 + 1π1 + 2π2 + 3π3 )2

n→∞

(c). (4 points) limn→∞ P {Xn = 0}.

lim P {Xn = 0} = π0 = 0
n→∞

(d). (4 points) Start the chain according to the initial distribution α(0) = π = (π0 , π1 , π2 , π3 ).
What is P {X5 = 2, X6 = 3}?
Since α(0) = π, we know that α(n) = π, for all n ≥ 0.
We compute
(5)
P (X5 = 2, X6 = 3) = P (X5 = 2)P (X6 = 3|X5 = 2) = α2 · P23 = π2 · P23 = (4/9)(1/2) = 2/9

4. (20 points) Consider a Markov chain on state space S = {0, 1, 2} with one-step transition
matrix
0 1 0
 

P = 1 − p 0 p
1 0 0
Here, p is a number between 0 and 1. Let Xn denote the state at time n.
(a). (8 points) Find E{(X7 )3 − 2X5 |X4 = 2}.

6
 E[(X7 )3 − 2X5 |X4 = 2] = E[(X7 )3 |X4 = 2] − E[2X5 |X4 = 2]
= [03 P20
3
+ 13 P21
3
+ 23 P22
3
] − [20 P20 + 21 P21 + 22 P22 ] = 8p − 1
(b). (12 points) Given that it is seven times more likely to be in state 0 than in state 2 (in the
long run), what is the value of p?
We are told that π0 = 7π2 .
The πi ’s are given by solving the following system of equations (π = πP , i πi = 1):
P

π0 = (1 − p)π1 + π2
π1 = π0
π2 = pπ1
1 = π0 + π1 + π2

Solving, we get π0 = π1 = 1/(2 + p) and π2 = p/(2 + p). Using the fact that π0 = 7π2 , we get that
p = 1/7.
5. (20 points) Consider the Markov chain illustrated in the transition diagram below. The state
space is the (infinite) set S = {A, B, C, D, E, F, G, H, I, 0, 1, 2, 3, . . .}. (For i ≥ 2, Pi,i+1 = Pi,i−1 =
1
2 .)

B C
F
H

1 1/4 1/2 1/2

E 0 1 2 3
D 3/4 1/2 1/2 1/2
G

I
A

(a). (5 points) List the communicating classes.

(b). (4 points) For each class, determine if it is transient or recurrent. For recurrent classes,
state whether each is positive recurrent or null recurrent.
(c). (5 points) For each class, determine its period.
{0, 1, 2, 3, . . .}: null recurrent, period=2
{A}: positive recurrent, period=1
{B}: positive recurrent, period=1
{C, D, E}: transient, period=3
{F, G, H, I}: transient, period=2
(d). (3 points) What is the probability that, starting in state C, the chain ever visits state 3?
0 (since there is no directed path from state C to state 3)
(e). (3 points) Find limn→∞ P {Xn ∈ {E, F, H}}.
0 (since states E, F, and H are each transient, the long-run probability that the chain is in any
one of these states is 0)

7
AMS 342 Joe Mitchell
Operations Research II: Stochastic Models: Yet Another Practice Midterm
1. (20 points) Assume that the number of telephone calls arriving at a switchboard by time t is
described by a Poisson process, {N (t), t ≥ 0}. We observe that, on average, one call comes in every
10 minutes.
(a). (4 points) What is the probability that no calls occur in the first ten minutes (0 < t ≤ 10
minutes) and that exactly one call occurs in the following five minutes (i.e., in the interval 10 <
t ≤ 15 minutes)?
(b). (4 points) What is the expected time at which the fourth call arrives?
(c). (4 points) What is the probability that two or more calls occur in the interval 10 < t ≤ 20?
(d). (4 points) Given that exactly one call arrived in 0 < t ≤ 15, calculate the (conditional)
probability that it occurred in 10 < t ≤ 15.
(e). (4 points) Given that exactly one call arrived in 0 < t ≤ 15, calculate the (conditional)
probability that at least one call will arrive in 15 < t ≤ 20.
2. (a). (10 points) A company has two machines. During any day, each machine that is working
at the beginning of the day has a 1/3 chance of breaking down. The two machines break down
independently of each other. If a machine breaks down during day n, it is sent to a repair facility
and will be working at the beginning of day n + 2. Thus, if a machine breaks down during day 3,
it will be working at the beginning of day 5. (The repair facility has enough repairmen that it can
work on both machines at once and still hold to the above schedule of returning the machines to
service.) Letting the state, Xn , of the system on day n be the number of machines working at the
beginning of day n, find the single-step transition matrix P for this Markov chain.

0 0 1
 

P =  0 1/3 2/3 
1/9 4/9 4/9
(b). (10 points) Consider the above problem (part (a) of 2) in which now we assume that if a
machine breaks down during day n, it will return to service at the beginning of day n + 3. If at any
time both machines are in the repair shop, the shop calls in a special super repairwoman who gets
them both working by the next morning. (For example, if on day n there is one machine already
in the shop and the other machine breaks down during day n, then both machines will be working
at the beginning of day n + 1.) Define a state space (be specific in describing the meaning of each
state!) and find the corresponding single-step probability transition matrix P for a Markov chain
that describes this new system.
Let S = {1A, 1B, 2}, where
1A if one machine is working at the beginning of day n, and the other machine


will be ready on the morning of day n + 1




Xn = 1B if one machine is working at the beginning of day n, and the other machine
will be ready on the morning of day n + 2





2 if both machines are working on the morning of day n
The transition matrix, with rows and columns labelled in order 1A, 1B, 2, is given by:
0 0 1
 

P =  2/3 0 1/3 
0 4/9 5/9

8
For example, we go from state 2 to state 2 if (i) both machines break down during day n (implying
that the super repairwoman has them both working by the morning of day n + 1), or if (ii) both
machines stay working during day n.
We go from state 2 to state 1B if one machine breaks down during day n, and the other machine
does not break down.
We go from state 1A to state 2 with certainty (probability 1), since, if the working machine
breaks down, then the super repairwoman gets both machines back in working order by the morning
of the next day.
3. (20 points) The New York Herald Tribune has obtained the following information about its sub-
scribers: During the first year as subscribers, 20 percent of all subscribers cancel their subscription.
Of those who have been subscribing for one year, 10 percent cancel during the second year. Of those
who have been subscribing for more than two years, 4 percent will cancel during any given year.
(a). (11 points) Model this process as a Markov chain. What is the state space S? Give the
exact meaning for being in state i ∈ S at time n. What is the single-step probability transition
matrix P ?
Let S = {C, 0, 1, 2+ }, where we think of Xn as the “state” of the subscription for a particular
customer, where
C if the customer has cancelled by year n



0 if the customer is new in year n
Xn =

 1
 +
if the customer has been a subscriber for exactly 1 year
2 if the customer has been a subscriber for ≥ 2 years

(We assume that a customer never resubscribes after cancelling.)

1 0 0 0
 
 1/5 0 4/5 0 
P =
 1/10

0 0 9/10 
1/25 0 0 24/25
The classes are {0} (transient), {1} (transient), {2+ } (transient), and {C} (positive recurrent).
(b). (9 points) On the average, how long does a subscriber subscribe to the New York Herald
Tribune? (i.e., if I subscribe at time 0, what is the expected value of the time when I cancel?)
Let τi = E[time until absorbed |X0 = i]. We want τ0 .
We condition on the first transition to get a system of equations:

τC = 0
τ0 = 1 + (1/5)τC + (4/5)τ1
τ1 = 1 + (1/10)τC + (9/10)τ2+
τ2+ = 1 + (1/25)τC + (24/25)τ2+

Solving, we get τ2+ = 25 years, τ1 = 23.5 years, and τ0 = 19.8 years. The expected length of a
subscription is τ0 = 19.8 years.
4. (20 points) Consider a Markov chain on state space S = {0, 1, 2, 3}. Let the vector π =
(π0 , π1 , π2 , π3 ) be the long-run distribution (i.e., the stationary distribution). Let Pijn be, as usual, the

9
n-step transition probability from state i to state j. (So, Pij is the single-step transition probability.)
Assume that the distribution of where the chain starts at time 0 is given by the vector α(0) = π
(i.e., start the chain according to the stationary distribution).
(a). (13 points) Compute the following quantities. Express your answers in terms of the πi ’s and
the Pijn ’s.
(i). P {X2 = 1} = π1
(ii). P {X5 = 0, X7 = 2} = π0 · P022

(iii). P {X5 = 0, X7 = 2 | X3 = 2} = P20 2 · P2

02
(iv). V ar[X3 ] = (0 π0 + 1 π1 + 2 π2 + 3 π3 ) − (0π0 + 1π1 + 2π2 + 3π3 )2 , since we know that X3
2 2 2 2

has distribution π, in this case, since the chain was started with α(0) = π.
(v). E[X5 | X3 = 2] = 0 · P20 2 + 1 · P2 + 2 · P2 + 3 · P2 .
21 22 23
(b). (7 points) A careless AMS 342 student spilled coffee on his homework paper. He was trying
to find the values of the π vector, and he knew the matrix P that gives the single-step transition
probabilities. The problem is that he has now lost the paper on which he wrote P , and he wants
to reconstruct P from the set of equations he wrote below. The symbol “??” means that the
corresponding number is not legible, due to the coffee spillage. What is the matrix P ? (As in (a),
the state space is S = {0, 1, 2, 3}.)

π0 = 14 π0 + 13 π1 + π2
π1 =??π0 +??π3
π2 = 21 π0 + 13 π1
π3 =??π1 + 21 π3
1 = π0 + π1 + π2 + π3

The equations above come from π = πP , i πi = 1. Thus, we know that

P

1/4 ? 1/2 0
 
 1/3 0 1/3 ? 
P =
 1

0 0 0 
0 ? 0 1/2
Now, we also know that the row sums of P are 1, so we get that

1/4 1/4 1/2 0

 
 1/3 0 1/3 1/3 
P =
 1

0 0 0 
0 1/2 0 1/2

5. (20 points) If today is a payday, then you receive a check for $1000 at 8:00am in the morning.
If today is not a payday, you receive no check. If today is a payday, then tomorrow will be a payday
with probability 1/3. If today is not a payday, then tomorrow will be a payday with probability 1/2.
(a). Write down a state space and transition matrix for the Markov chain that models the above
process.
We define the state space to be S = {0, 1}, where
0 if day n is not a payday

Xn =
1 if day n is a payday

10
Then, the single-step transition probability matrix is

1/2 1/2
 
P =
2/3 1/3

We know that the Markov chain is irreducible, positive recurrent, aperiodic; it is ergodic.
(b). Today is February 28, 2007. Estimate the probability that you receive a paycheck on the
morning of December 28, 2007.
Suppose today is day 0 and let day n be December 25, 2004, which is many days away (over
300) from now. We want to compute
(n)
P (Xn = 1) = α1 ≈ lim P (Xn = 1) = π1
n→∞

Thus, we want the π vector satisfying

X
π = πP, πi = 1,
i

which gives the system of equations

π0 = (1/2)π0 + (2/3)π1
π1 = (1/2)π0 + (1/3)π1
π0 + π1 = 1,

which solves to π0 = 4/7, π1 = 3/7. Thus P (Xn = 1) ≈ 3/7.

(c). You just received a paycheck. On average, how long must you wait until your next paycheck?
We want the expected return time to state 1, which is m1 = 1/π1 = 7/3 days.

11