Markov Chain

A Markov process with discrete state space is called a Markov Chain.

(According to some authors, e.g., U. N. Bhat, a Markov process with discrete time
is called Markov Chain.)
Example: The process in Example 1 (gambling) is a Markov Chain, because (i) it
has Markov property and (ii) the state space is discrete. It is a discrete-time Markov
Chain.
Example: (Customer arrival) Let 𝑋𝑋(𝑡𝑡) = Number of customers coming to a bank in
(0, 𝑡𝑡]. Then, 𝑋𝑋(𝑡𝑡) has Markov property, because, for example, if 𝑋𝑋(3.5) is known,
𝑋𝑋(4) does not depend on 𝑋𝑋(2.5). Here, 𝑋𝑋(𝑡𝑡) is a continuous-time Markov Chain.

Discrete-time Markov Chains

Transition probability
Let
(𝑚𝑚,𝑛𝑛)
𝑝𝑝𝑖𝑖𝑖𝑖 = 𝑃𝑃(𝑋𝑋𝑛𝑛 = 𝑗𝑗 | 𝑋𝑋𝑚𝑚 = 𝑖𝑖); 𝑖𝑖 ∈ 𝑆𝑆, 𝑗𝑗 ∈ 𝑆𝑆

be the conditional probability that the process will have transition to state 𝑗𝑗 at time
𝑛𝑛 given that it was in state 𝑖𝑖 at time 𝑚𝑚.
Example: (Gambling) If the coin is fair
(2,4) 1
𝑝𝑝31 = 𝑃𝑃(𝑋𝑋4 = 1 | 𝑋𝑋2 = 3) =
4
When 𝑛𝑛 = 𝑚𝑚 + 1, we have
(𝑚𝑚,𝑚𝑚+1)
𝑝𝑝𝑖𝑖𝑖𝑖 = 𝑃𝑃(𝑋𝑋𝑚𝑚+1 = 𝑗𝑗 | 𝑋𝑋𝑚𝑚 = 𝑖𝑖); 𝑖𝑖 ∈ 𝑆𝑆, 𝑗𝑗 ∈ 𝑆𝑆

which is called one-step transition probability at time 𝑚𝑚.

Time-homogeneous Markov Chain

(𝑚𝑚,𝑚𝑚+1)
When 𝑝𝑝𝑖𝑖𝑖𝑖 is independent of 𝑚𝑚, we say that the Markov Chain is time-
homogeneous. In that case, we drop 𝑚𝑚 and write the one-step transition probability
(1)
as 𝑝𝑝𝑖𝑖𝑖𝑖 = 𝑝𝑝𝑖𝑖𝑖𝑖 .

3
Also, the 𝑛𝑛-step transition probability for a time-homogeneous Markov Chain can
(𝑚𝑚,𝑚𝑚+𝑛𝑛) (𝑛𝑛)
be written as 𝑝𝑝𝑖𝑖𝑖𝑖 = 𝑝𝑝𝑖𝑖𝑖𝑖 . For 𝑛𝑛 ≥ 2, we cannot drop 𝑛𝑛.

Example: (Gambling)
(2,3) 1
𝑝𝑝34 = 𝑃𝑃(𝑋𝑋3 = 4 | 𝑋𝑋2 = 3) =
2
(4,5) 1
𝑝𝑝34 = 𝑃𝑃(𝑋𝑋5 = 4 | 𝑋𝑋4 = 3) =
2
Therefore, the chain is time-homogeneous. We can write both the probabilities as
(1)
𝑝𝑝34 = 𝑝𝑝34 .

Time-nonhomogeneous Markov Chain

Example:
Let us consider marital status of a man over discrete time, where time is age in years.
Let
𝑆𝑆 = {Unmarried, Married, Divorced, Widowed, Remarried}.
(21,22)
𝑝𝑝𝑈𝑈𝑈𝑈 = 0.05 (say)
(26,27)
𝑝𝑝𝑈𝑈𝑈𝑈 = 0.10
(31,32)
𝑝𝑝𝑈𝑈𝑈𝑈 = 0.25
We see that, transition probability changes over time.
Example:
Let 𝑆𝑆 = {𝐻𝐻, 𝐷𝐷}, where H means healthy and D means diseased.
(19,20)
𝑝𝑝𝐻𝐻𝐻𝐻 = 0.02 (say)
(60,61)
𝑝𝑝𝐻𝐻𝐻𝐻 = 0.30
(1)
We cannot write 𝑝𝑝𝐻𝐻𝐻𝐻 because at 19 and 60, the probabilities of transition from
healthy to disease are different.