Appendix to Lecture 11, Examples The following example shows that the value function V may not be continuous if we do not assume that G is lower semicontinuous, Example: (Discontinuous value funetion) Let X= Cy ¥) © REL (y, 9)? + (y,-8)* 82) As a step in the definition of G, let P={(¥s¥ 2,2) [yy ¥,) © XU(4, 4, 6, 0) 12 sos 4) Let the graph of G be the convex hull of , which is the set of all convex combinations of points in. The graph of G Is convex and compact. The correspondence G is defined by the equation yey) yy ¥) LO Yp ye YQ. © GFapH of G} Let A= XxX and let u: XxX = R be defined by the equation ux x») Se and let B ¢ (0, 1). The set G(x) is not hard to visualize. Let C be the circle that is the boundary of X. That is, 3)? = 2), C= Uy y) ERE (y,- 9)? #0, sothat (4,4) eC. Ix ©X and x # (4, 4), there is @ unique line segment going from (4, 4) through x. Let a(x) be the intersection of C with this line segment that is distinct from (4, 4). Then x Is a convex combination of a(x) and (4, 4). Let a(x) be defined by the equation, x = a(xjax) + (1 — a(x))(4, 4). Let a(4, 4) = 0. Notice that a(x) = 1, if x eC and x ¥ (4, 4). Then Qx) ={(r, ) [2 81s 4-2a(x)} ©X, By lemma 5.4, G is upper semicontinuous, because its graph is a closed set and for all x, G(x) is fa subset of the compact sot X. G(x) is not empty, for all x. The sets X and A are compact and u is continuous. The sets A and X and the graph of G are convex, X has non-emply interior, and u is concave, so that the example satisfies assumptions 1 - 4, 8, and 9. | show that the correspondence G is not lower semicontinuous at the point (4, 4). Let x" be a sequence in C such that x" # (4, 4), for all1n, and lim x"= (4, 4). Then @x") = ((2, 2)}, for all n, andG(4, 4)) = {(r, | 2< <4}. Hence there is no sequence y" such that y® © G(x") , for all n, andlim y*= (4, 4), and so G is not lower semicontinuous at (4, 4) and the example does not satisfy assumption 5. The next figure illustrates the definition of a(x). 1 2 3 4 It x, = (4, 4) then itis optimal to choose x, = (4, 4) and enjoy utility & in period o. ‘The optimal choice is then repeated indefinitely and a utility of 8 is enjoyed in every period. Therefore V(4, 4) = 8/(1 ~ p). Ix, is any point in the circle C other than (4, 4), then the only choice possible is (2, 2), which is in G and not equal to (4, 4), and so (2, 2) must be chosen in period 1 and in every period thereafter. Therefore V(x ) = 4/(1—B) , and so the value function is not continuous at (4, 4). This ends the discussion of the example. The next example shows that Blackwell’s theorem does not apply to every contraction mapping. Example: Let Z = ((n,,m,) € RP] m 4, = 1). That is, Z is the set of probability vectors over two events. Let T:Z + Z be defined by= = (4 173)", TH 8) = Uva 20a he, Ix = (x. x) © RF, let the norm of x be Ix] = max(|x |, |x,|). That is, the norm of x is the epee & ely ol+ » maximum of the absolute values of the components of x. Ifm = (x, m,) andy =(n,,,) belong (M, -M), for some number M. to Z, then x = ~ has the property that x, + x, = 0, 80 that x 5_(M, -M). Therefore T( = 5_(n x), for 12 12 nand xin Z. Therefore [T(n - x) | = §_|(1 - x) | and so Tis a contraction on C. Itis easy to 12 check algebraically that (¢- 4) is the unique fixed point of T. Blackwell's theorem does not 77 apply to T, however. An unimportant reason why it does not apply is that ifm = (n,m) © Zand It is not hard to calculate that T(M, -M) a is a non-negative number, then (x, +a, x, + a) does not belong to Z. This alone should not matter, however, because T(x, + a,x, + a) is still defined. The real difficulty is that 12 12 number B such that 0 < B <1. That is, T does not satisfy condition b of Blackwell's theorem. This ends the discussion of the example. Tn + (a, a)) = Tn) + (2 its} which is not less than or equal to T(r) + Bla, a), for any ‘The matrix of the linear transformation T in the previous example is an example of a transition matrix of a Markov process. If the states are 1 and 2, then and, are the probabilities of states 1 and 2. If the state currently is 1, the probability that in the next period the state will be 1 is 3/4 and the probability that the state will be 2 is 1/4. Similarly if the state currently is 2, the probabilities that in the next period the state will be 1 or 2 are 1/3 and 2/3, respectively. The probability distribution x = (n,m) = (4/7, 3/7) is called a stationary distribution, because it is preserved after the random transition ‘A Markov transition over N states, n = 1, 2, ...., N, Is defined by an NxN matrix A with (k, n)th entry a, where @ is the probability that the state will be k in the next period if it is nin the current period. Then a 20, forall k and n, and )) @ = 1, for all n, so that the columns of A are probability vectors. if the probability distribution over the states currently then the probability the state is k in the succeeding period is). aw isn=(n,x, The vector of these probabilities is(hay : Ban) or in vertical form " = An. Lan The probability distribution over states two periods hence is A’z, and the probability distribution t periods from now is A'x, where A is A to the power t. The next theorem asserts that as t goes to infinity, this distribution converges to a unique stationary distribution such that An =n, provided a > 0, for all k and n. | prove the theorem, because the demonstration is ‘a contraction argument somewhat different from that of theorems 11.4 or 11.7. » for all n, and ‘Theorem 11.23: Let A be a non-negative NXN matrix such that J) a a> 0, forall kandn. Then B=lim A' exists and is such that if b, is the kth row of B, for k = m w : Trowey Ne then b= (mo ny ow oth), where n, > Oand Y= 1. tn=|* } then h An =n. If is any probability distribution over the N states, then lim A'x =n, Proof: Let x! = (x!., x!) be the kK row of A’. Then x! = 0, for all, k, and n. Let ME = max(x! ya x8) and fet m! = min(x! x4). Lete = min a, which by assumption is @ positive number. Without loss of generality, assume that m! = x! . Then re xa axa + Dota = ma + Ex sma, +M Sa ma + M1 =a) sem'4 (1~e) Ml.Since MI! = max(x!*", (1) , we see that MIT sem! + (12) Mt A similar argument proves that mi" > eM! + (1 ~e) mt Hence Met Ts em! 4 (1 ~e) M\~ eM! ~ (1 ~e)m! = (1 ~2e) (Mim), so that Mi = mis (1 ~2e)"*(M! ml), for all t, and hence lim (Mim) = 0 (11.17) ‘The sequence M! is non-negative and non-inereasing in t, and m! is bounded above by M! and is non-decreasing in t, so that both sequences converge by the monotone convergence theorem 4.17. Inequality 11.17 implies that they must converge to the same limit. Call that limit ,. Sincem'= min a m At equals the positive number n, > 0 andn, 2 m', we know that n, > 0. So every entry of the kth row of B= . Lote be the N-vector all of the components of which are 1. Since U show that Fn, the columns of A are probability vectors, 6'A~= e". Therefore et? = e'AA = e'A=e! (n Continuing by induction on t, we see that e"A' = e", for all. Let =| ° } Then a(iimay eM(n, m,n) and so 1 ef =e. Thatis, Yn, I show that An =n, Notice that AB = A(lim aye lim A‘ = B. Hence ifb! is the first column of B, Abt = b'. Since b= n, An =n.| show that lim A'n = n, ifr is any probability distribution over the N states. Let b, be the (k, nh entry of B. Since b= n,, forallkandn, bx = Ene =n Ya =n, Hence Br =n and so lim An = Bx = .