Lecture 6 : Markov Chains

STAT 150 Spring 2006 Lecturer: Jim Pitman Scribe: Alex Michalka <>

Markov Chains Discrete time Discrete (nite or countable) state space S Process {Xn } Homogenous transition probabilities matrix P = {P (i, j ); i, j S } P (i, j ), the (i, j )th entry of the matrix P , represents the probability of moving to state j given that the chain is currently in state i. Markov Property: P(Xn+1 = in+1 |Xn = in , . . . , X0 = i0 ) = P (in , in+1 ) This means that the states of Xn1 . . . X0 dont matter. The transition probabilities only depend on the current state of the process. So, P(Xn+1 = in+1 |Xn = in , . . . , X0 = i0 ) = P(Xn+1 = in+1 |Xn = in ) = P (in , in+1 ) To calculate the probability of a path, multiply the desired transition probabilities: P(X0 = i0 , X1 = i1 , X2 = i2 , . . . , Xn = in ) = P (i0 , i1 ) P (i1 , i2 ) . . . P (in1 , in ) Example: iid Sequence {Xn }, P(Xn = j ) = p(j ), and
j

p(j ) = 1.

P (i, j ) = j. Example: Random Walk. S = Z (integers), Xn = i0 + D1 + D2 + . . . + Dn , where Di are iid, and P(Di = j ) = p(j ). P (i, j ) = p(j i). Example: Same random walk, but stops at p(j i) 1 P (i, j ) = 0 6-1 0. if i = 0; if i = j = 0; if i = 0, j = 0.

Lecture 6: Markov Chains

6-2

Example: Y0 , Y1 , Y2 . . . iid. P(Y = j ) = p(j ). Xn = max(Y0 , . . . , Yn ). if i j ; p(j ) p ( k ) if i = j; P (i, j ) = k i 0 if i > j. Matrix Properties: P (i, ) = [P (i, j ); j S ], the ith row of the matrix P , is a row vector. Each row of P is a probability distribution on the state-space S , representing the probabilities of transitions out of state i. Each row should sum to 1. ( P (i, j ) = 1 i )
j S

The column vectors [P (, j ); j S ] represent the probability of moving into state j . Use the notation Pi = probability for a Markov Chain that started in state i (X0 = i). In our transition matrix notation, Pi (X1 = j ) = P (i, j ). n-step transition probabilities: Want to nd Pi (Xn = j ) for n = 1, 2, ... This is the probability that the Markov Chain is in state j after n steps, given that it started in state i. First, let n = 2. Pi (X2 = k ) =
j S

Pi (Xi = j, X2 = k ) P (i, j )P (j, k ).

j S

= This is simply matrix multiplication.

So, Pi (X2 = k ) = P 2 (i, k ) We can generalize this fact for n-step probabilities, to get: Pi (Xn = k ) = P n (i, k ) Where P n (i, k ) is the (i, k )th entry of P n , the transition matrix multiplied by itself n times. This is a handy formula, but as n gets large, P n gets increasingly dicult and time-consuming to compute. This motivates theory for large values of n. Suppose we have X0 , where is a probability distribution on S , so P(X0 = i) =

Lecture 6: Markov Chains

6-3

(i) for i S. We can nd nth step probabilities by conditioning on X0 . P(Xn = j ) =

iS

P(X0 = i) P(Xn = j |X0 = i) (i)P n (i, j )

iS

= (P n )j = the j th entry of P n . Here, we are regarding the probability distribution on S as a vector indexed by i S. So, we have X1 P, X2 P 2 , . . . , Xn P n . Note: To compute the distribution of Xn for a particular , it is not necessary to nd P n (i, j ) for all i, j, n. In fact, there are very few examples where P n (i, j ) can be computed explicitly. Often, P has certain special initial distributions so that computing P n is fairly simple. Example: Death and Immigration Process Xn = the number of individuals in the population at time (or generation) n. S = {0, 1, 2, . . .} Idea: between times n and n + 1, each of the Xn individuals dies with probability p, and the survivors contribute to generation Xn+1 . Also, immigrants arrive each generation, following a Poisson() distribution. Theory: What is the transition matrix, P , for this process? To nd it, we condition on the number of survivors to obtain:
ij

P (i, j ) =
k =0

i j

(1 p)k (p)ik

e j k (j k )!

Here, i j = min{i, j }. This formula cannot be simplied in any useful way, but we can analyze the behavior for large n using knowledge of the Poisson/Binomial relationship. We start by considering a special initial distribution for X0 . Let X0 Poisson(0 ). Then the survivors of X0 have a Poisson(0 q ) distribution, where q = 1 p. Since the number of immigrants each generation follows a Poisson() distribution, independent of the number of people currently in the population, we have X1 Poisson(0 q + ). We can repeat this logic to nd: X2 Poisson((0 q + )q + ) = Poisson(0 q 2 + q + ), and

Lecture 6: Markov Chains

6-4

X3 Poisson((0 q 2 + q + )q + ) = Poisson(0 q 3 + q 2 + q + ). In general, Xn Poisson(0 q n +

n1 k =0

q k ).

In this formula, 0 q n represents the survivors from the initial population, q n1 represents the survivors from the rst immigration, and so on, until q represents the survivors from the previous immigration, and represents the immigrants in the current generation. Now well look at what happens as n gets large. As we let n : 0 q n 0, and 0 q n +
n1 k =0

qk

1q

= . p

So, no matter what 0 , if X0 has Poisson distribution with mean 0 , limn P(Xn = k ) =
e k , k!

where =

1q

= . p

It is easy enough to show that this is true no matter what the distribution of X0 . The particular choice of initial distribution for X0 that is Poisson with mean 0 = gives an invariant (also called stationary, or equilibrium, or steady-state) distribution of X0 . With 0 = , we nd that each Xn will follow a Poisson( ) distribution. This i initial distribution (i) = e i! is special because it has the property that (i)P (i, j ) = (j ) for all j S .
iS

Or, in matrix form, P = . It can be shown that this is the unique stationary probability distribution for this Markov Chain.