OPERATIONS RESEARCH

Markov Chains

Dr. K. K. Pandey 1-1

Introduction

Markov chain models are a particular class of probabilistic models known

as stochastic processes, in which the current state of a system depends
on all of its previous states. But in many problems, a sequence of events
or outcomes is considered to be independent. That is, any given event or
outcome does not depend on any of the preceding events or outcomes.
A more general model assumes that there is a one-stage dependence of
events, with each event depends only on the immediately preceding
event, but not on other prior events. A process (sequence) of the type is
said to be a Markov chain process, Markov chain or Markov process.

1-2
For example, consider the following few systems.
• Market place for a product and its competitive brands.
• Machines used to manufacture a product.
• Billing, credit and collection procedures involved in converting
accounts receivable from the product’s sales into cash.
• Area of specialization by a management student at the same time.

Each system, described above, may be in one of several possible states.

These states describe all possible conditions of a system (or process):
• In marketing example, states may be expressed in terms of the brand
that a customer is presently using.
• In production example, a machine can be in one of two states:
Working or not at any point in time.
• In financial transaction example, the accounts receivable can fall into
one of the states: Cash sale, Credit sale or Incollectable money.
• A student can specialize in only few management functional areas
and not in all areas at the same time.
https://analyticsindiamag.com/5-real-world-use-cases-of-the-markov-chains/\ 1-3
Characteristics of a Markov Chain

• There are finite number of possible states.

• States are both collectively exhaustive and mutually exclusive.
• The transition probabilities depend only on the current state of the
system, i.e. if current state is known, the conditional probability of the
next state is independent of the states prior to the present state.
• The long-run probability of being in a particular state will be constant
over time.
• The transition probabilities of moving to alternative states in the next
time period, given a state in the current time period must sum to 1.0.

1-4
Applications of Markov Analysis
■ Personnel:  Determining future manpower requirements of an
organization taking into consideration retirements, deaths,
resignations, etc.
■ Finance:  Customer accounts receivable behaviour.
■ Production:  Helpful in evaluating alternative maintenance policies,
certain classes of inventory and queuing problems, inspection and
replacement analysis.
■ Marketing:  Useful in analysing and predicting customer’s buying
behaviour in terms of loyalty to a particular product brand, switching
patterns to other brands, and market share of the company versus its
competitors.

1-5
State and Transition Probabilities
Predicting future states of any system requires the knowledge of the probability
of changing from one state to another. These probabilities are arranged in a
matrix, called the matrix of transition probabilities and shows the likelihood that
the system will change from one state time period to the next.

In general, let

Si = finite number of possible outcomes (i=1, 2, ..., m) of each of the sequence

of experiments e.g., purchases and possible outcomes are three brands of
the product)
M = number of states
Pij = conditional probability of outcome sj for any particular experiment given
that outcome si occurred for the immediately preceding experiment (or
event), That is, probability of being in state j in the future given the current
state i.

1-6
. . . state and transition probabilities

The outcomes s1, s2, . . ., sm are called states and the numbers pij are called
transition probabilities.

Suppose the process begins in some particular state, then it is easy to

calculate probability of states relating to the overall sequence of events.

Each time a new state is reached, the system is said to have stepped (or
incremented) one step ahead. Each step represents a time period (or
condition) which results in another possible state.

1-7
. . . state and transition probabilities

If pij are assumed constant, (do not change from one event to another of the
sequence over time), then the Markov chain is said to be stationary,
otherwise said to be non-stationary or time dependent. For a Markov chain
with states s1, s2, ..., sm, the matrix of transition probabilities is written as:

 Succeeding state
s1 s2         sm
S1 p11 p12 … p1m
P = [ pij]m × m = Initial state S2 p21 p22 … p2m
.
. . . .
. . . . .
. . . .
.
sm pm2 pm2 pmm

https://setosa.io/ev/markov-chains/

1-8
. . . state and transition probabilities

If pij = 0, then no transition occurs from state i to state j. But if pij = 1, then the
system is in state i, and can move only to state j at the next transition.

The sum of the elements in each row of the matrix P is one because these
elements represent probabilities of all transitions when the process is in
state si, i.e.
m
o pij ∑ 1,pijfor= all
1, i
j= 1

1-9
Transition diagram

The transition probabilities can also be represented by the transition diagram

where arrows from each state indicate the possible transitions to a start and
their corresponding probabilities. The matrix
of transition probabilities is written as:

 Succeeding state
s1 s2      s3
s1 0 p12 0
P = Initial state s2s 0 p22 p23
p31 0 p33
3

Transition Diagram
A zero element in the transition matrix indicates
that the transition is impossible.

1-10
Probability Tree Diagram
These diagrams are used to evaluate and determine the probability that the
given system will be in any particular state at any particular time, given the
current state.

The number in circles in the tree diagram represent the state at the beginning of
a transition.

Probabilities of shifting from s1 to s1 itself and s2 as well as from state s2 to s1

and s2 itself are represented as elements of a transition matrix as follows:
State 2
s1 s2 s1 s2
s1 p11 p12 s1 0.7 0.3
P = state 1 =
s2 p12 p22 s2 0.8 0.2

Here, p11 + p12 = 1 and p21 + p22 = 1

1-11
Tree Diagram

1-12
Multi-Period Transition Probabilities

Given vector of state probabilities, R0 = [P11, P12, . . ., P1m] and the matrix
of transition probabilities, at time period n = 0, we can determine the
state probabilities at a future time. For convenience, let R1 represents the
state probabilities at time (or state) n = 1. After one execution of the
experiment it can be written in terms of row matrix as:

R1 = R0 × P

1-13
. . . multi-period transition probabilities

The vector of state probabilities at time (or state) 2, 3, . . ., n, are obtained
by multiplying the system state at time 0 with the transition martix (P),
that is
R2 = R1 × P = R0 × P2

Rn = Rn – 1 × P = R0 × P n

The elements of the n-step transition matrix Pn = [pnij]m × m are obtained by

repeatedly multiplying the transition matrix P by itself. In general Pn =
P n–1 P
where each row i of Pn represents the state probability distribution after n
transitions, given that the process starts out in state i.

1-14
Example: Two manufacturers A and B are competing with each other in a
restricted market. Over the year, A’s customers have exhibited a high
degree of loyalty as measured by the fact that customers are using A’s
product 80 per cent of the time. Also former customers purchasing the
product from B have switched back to A’s product 60 per cent of the time.

• Construct and interpret the state transition matrix in terms of

(i) retention and loss, and (ii) retention and gain.

• Calculate the probability of a customer purchasing A’s product

at the end of the second period.

1-15
Solution: The transition probabilities can be arranged in matrix form as
follows. The probability of a customer’s purchase at the next step n=1
(next purchase) depends upon the product which a customer is presently
having now, i.e., at step n = 0 (present purchase). Each probability in the
following matrix must, therefore, be a conditional probability for going from
one state to another.

Next purchase
(n = 1)
A      B
  A 0.80 0.20
P=  
B 0.60 0.40
Retention and loss

1-16
Probability Calculations:  Starting with a customer’s purchase of A’s
product in state s1 at n = 0, then p11 = 1 and p12 = 0, so R0  =  [1    0]

After the first transition, the state probabilities R1 which describes all
possible outcomes at n = 1 is given by

0.80 0.20
R1 = R0 × P = [1    0]    0.60 0.40
= [0.80    0.20]

Transition Diagram

1-17
Probability Tree Diagram

1-18
The probability that a customer using A’s product in the state s1, at n = 0,
also use A’s product in the state s2 at n = 2 is obtained as follows:

R2 = R1 × P
0.80 0.20
= [0.80    0.20] 0.60 0.40= [0.76   0.24]

This suggests that if the present state is s1 at n = 0, two periods later (i.e. at
n = 2) the probability of being in state s1 is = 0.76 and in state s2 is = 0.24.
Hence, the probability of A’s market share after the end of two periods is 76
per cent and that of B’s market share is 24 per cent.

1-19
Example:   There are three dairies A, B and C in a small town which
supply all the milk consumed in the town. Assume that the initial consumer
sample is composed of 1,000 respondents distributed over three dairies A,
B and C. It is known by all the dairies that consumers switch from one dairy
to another due to advertising, price and dissatisfaction. All these dairies
maintain records of the number of their customers and the dairy from which
they obtained each new customer. The following table illustrates the flow of
customers over an observation period of one month. Assume that the
matrix of transition probabilities remains fairly stable and at the beginning
of period one, market shares are A = 25%, B = 45% and C = 30%.

1-20
Construct the state transition probability matrix to analyse the problem.

Diary Period 1 Change during Period Period 2

(Customers) Gain Loss (Customers)

A 250 62 50 262
B 450 53 60 443
C 300 50 55 295

1,000  165  165  1,000 

1-21
Determine Retention Probabilities: To determine the retention probability of
cutomers, the customers retained for the period under review are divided by
the number of customers at beginning of the period. The results are
summarized below:

Dairy Period 1 Number Lost Number Retained Probability of

(Customers) Retention

A 250 50 200 200/250 = 0.800

B 450 60 390 390/450 = 0.866

C 300 55 245 245/300 = 0.816

1-22
 Gains and Losses under Review Period0

Dairy Period 1 Gains from Losses to Period 2

(Customers) A B C A B C (Customers)

A 250 10 35 27 10 25 25 262
B 450 25 10 28 35 10 25 443
C 300 25 25 10 27 28 10 295

A B C

A 200/250 = 0.80 25/250 = 0.10 25/250 = 0.10

Retention
B 35/450 = 0.80 390/450 = 0.866 25/450 = 0.055 and
Gain
C 27/300 = 0.09 28/300 = 0.093 245/300 = 0.81
— Retention and loss →

1-23
Steady-state (Equilibrium) Conditions
As the number of periods increase, further changes in the state probabilities
are smaller. Similarly, as the number of stages or transitions approaches
infinity, a Markov chain approaches a steady or equilibrium state in which the
probability distribution of its states become stationary. Thus, in the steady-
state the probability pi that a Markov chain is in any particular state, si, is
constant from trail to trail. The Markov chain reaches the steady-state
condition only when following conditions are met:

• The transition matrix elements remain positive from one period to the
next. This is referred to as the regular property of a Markov chain.
• It is possible to go from one state to another in a finite number of steps,
regardless of the present state. This is referred to as the ergodic (or
absorbing states) property of a Markov chain.

1-24
Example: On January 1 (this year), Bakery A had 40 per cent of its local
market share while the other two bakeries B and C had 40 per cent and 20 per
cent, respectively, of the market share. Based upon a study by a marketing
research firm, the following facts were compiled. Bakery A retains 90 per cent
of its customers while gaining 5 per cent of B’s customers and 10 per cent of
C’s customers. Bakery B retains 85 per cent of its customers while gaining 5
per cent of A’s customers and 7 per cent of C’s customers. Bakery C retains
83 per cent of its customers and gains 5 per cent of A’s customers and 10 per
cent of B’s customers. What will each firm’s share be on January 1 next year
and what will each firm’s market share be at equilibrium?

1-25
Solution  Using the data of the given problem, the state transition matrix is written
as:
Bakery
                                       A          B               C
A Retention
0.90 0.05 0.05
Bakery B 0.05 0.85 and
0.10
Gain
C 0.10 0.07 0.83

— Retention and loss

1-26
The probable market shares for Bakery A, B and C on January 1 next year is
obtained by multiplying the state transition matrix by the market shares on
January 1, this year.

[Market share on = [Market share on Transition matrix]

January 1 (next year)] January 1 (this year)]
0.90 0.05 0.05
= [0.40    0.40    0.20] 0.05 0.85 0.10
0.10 0.07 0.83
= [0.400    0.374    0.226]

This implies that market shares of Bakeries A, B and C on January 1 (next

year) will be 40 per cent, 37.4 per cent and 22.6 per cent, respectively.

1-27
To calculate the percentage market for each Bakery at equilibrium, state
transition matrix is expressed as:
0.90 0.05 0.05
 =  0.05 0.85 0.10
0.10 0.07 0.83
Period n share Period n – 1 share Transition matrix

The market share in period n – 1 are the same as the shares in period n
because a steady-state condition has been reached.

1-28
Multiplying the two matrices (applying row by column multiplication rule) together
and setting each value equal to the share in period n, to get the following set of
equations. The last equation is based on the fact that the shares in any period
must total unity.

SA = 0.90 SA + 0.05 SB + 0.10 SC

SB = 0.05 SA + 0.85 SB + 0.07 SC

SC = 0.05 SA + 0.10 SB + 0.83 SC

SA  + SB + SC = 1

1-29
The values for the steady-state shares, SA, Sb and SC is obtained by solving
these equations simultaneously.

Rearranging first three equations, we get

– 0.10 SA + 0.05 SB + 0.10 SC = 0

– 0.05 SA – 0.15 SB + 0.07 SC = 0

– 0.05 SA + 0.10 SB – 0.17 SC = 0
– SA + SB + SC = 1
Subtracting third equation from second gives us the following result
0.05 SA – 0.15 SB + 0.07 SC ) = 0
– (0.05 SA + 0.10 SB – 0.17 SC = 0

– 0.25 SB + 0.24 SC = 0

or SB = (24/25) SC = 0.96 SC

1-30
Using the same pair of second and third equation, we can multiply second by 2 and
third by 3 and then add.
0.10 SA – 0.30 SB + 0.14 SC ) =0
+ ( 0.15 SA + 0.30 SB – 0.51 SC ) =0
0.25 SA + 0.30 SB – 0.37 SC  ) =0

or                 SA = (37/25) SC = 1.48 SC

Substituting these values into the last equation to solve for SC, we get
1.48 SC + 0.96 SC + SC =1
3.44 SC = 1  or  SC = 0.29,
Then SB = 0.96, SC = 0.96 (0.29) = 0.28
SA =  1.48 SC = 1.48 (0.29) = 0.43

Hence, the solution for equilibrium or steady-state market share are:

Bakery A = 43%; Bakery B = 28%; Bakery C = 29%.
1-31
1-32
1-33
1-34
1-35
1-36
1-37
1-38
1-39
1-40
1-41
1-42
1-43
1-44
1-45