Dynamic Programming and Optimal Control Volume I THIRD EDITION Dimitri P. Bertsekas “Massachusetts Institute of TechnologyAthena Scientific Post Office Box 805 Nashua, NH 03061-0805 USA, WWW: http://www.athenase.com Cover Design: Ann Gallager, wow.gallagerdesign.com © 2005, 2000, 1995. Dimitri P, Bertsekas All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, oo information storage and retrieval) without, perinissio the publisher Publisher's Cataloging-in-Publication Data. Bertsekas, Dimite P. Dynamic Programming and Optimal Control Includes Bibliography and Index |. Mathematical Optimization. 2, Dynamic Programming. I, Title. QA402.5 8465 2005 519.703 O0-91281 ISBN 1-886529-26-4 i i | | | j } | | | | | | i j | | ABOUT THE AUTHOR Dimitri Bertselass studied Mechanical and Bleetrical Engineering at the National Technical University of Athens, Greece, and obtained his Ph.D. in system science from the Massachusetts Institute of Technology. He has held faculty positions with the Engineering-Meonomic Systems Dept., Stanford University, and the Blectrical Engineering Dept. of the Univer sity of Ilinois, Urbana, Since 1979 he has been teaching at the Electrical Engineering and Computer Science Departinent of the Massachusetts In- stitute of Technology (M.LT:), where he is currently MeAfee Professor of Engineering. is research spans several fields, including optimization, control, large- scale computation, and data communication networks, and is closely tied to his teaching and book authoring activities, He has written munetous research papers, and thirteen books, several of whieh are used as textbooks in MIT classes. He consults regularly with private industry and has held editorial positions in several journals, Professor Bertsekas was awarded the INFORMS 1997 Prize for Re search Excellence in the Interface Between Operations Research aud Com puter Science for his book “Neuro-Dynamic Programming” (co-authored with John Tsitsiklis), the 2000 Greek National Award for Operations Re- search, and the 2001 ACC John R. Ragazzini Education Award, In 2001, hhe was elected to the United States National Academy of Engineering.6, ATHENA SCIENTIFIC OPTIMIZATION AND COMPUTATION SERIES Convex Analysis and Optimization, by Dimitri P. Bertselas, with Angelia Nedié and Asuman B, Ozdaglar, 2003, ISBN 1-886529- 45-0, 560 pages Introduction to Probability, by Dimitri P. Bertselas and John N. ‘Tsitsildis, 2002, ISBN 1-886520-40-X, 430 pages Dynamie Programming and Optimal Control, ‘Two-Volume Set, by Dimitri P, Bertsekas, 2005, ISBIN 1-886529-08-6, 810 pages Nonlinear Programming, 2nd Edition, by Dimitri P. Bertsekas, 1999, ISBN 1-886520.00-0, 791 pages Network Optimization: Continions and Diserete Models, by Din- itv P, Bertsekas, 1998, ISBN 1-886529-02-7, 608 pages Network Flows and Monotropic Optimization, by R. ‘Tyrrell Rock- ‘fellar, 1998, ISBN 1-886529-06-X, 634 pages Introduction to Linear Optimization, by Dimitris Bertsimas and John N. Tsitsiklis, 1997, ISBN 1-886529-19-1, 608 pages avallel and Distributed Computation: Numerical Methods, by Dimitri P. Bertselas and John N. Tsitsiktis, 1997, ISBN 1-886529- 01-9, 718 pages Neuto-Dynanie Programming, by Dimitri P, Bertsekas and John, N, Taitsiklis, 1996, ISBN 1-886529-10-8, 512 pages Constrained Optimization and Lagrange Multiplier Methods, by Dimitei P. Bertsekas, 1996, ISBN 1-886520-04-3, 410 pages Stochastic Optimal Control: The Diserote“Time Case, by Dimitri P. Rertselas and Steven B. Shreve, 1996, ISBN 1-886520-013 330 pages i | | | | i Contents 1. The Dynamic Programming Algorithm LAL. Introduction 1.2, The Basie Problem 1.8. ‘The Dynamic Programming Algorithm A. State Augmentation and Other Reformulations 1.5. Some Mathematical Issues 1.6. Dynamic Programming and Minimax Control LT. Notes, Sources, and Pxercises 2, Deterministic Systems and the Shortest Path Problem 2.1. Finite-State Systems and Shortest Paths 2.2. Sone Shortest Path Applications 2.2.1. Critieal Path Analysis 2.2.2. Hidden Markov Models and the Viterbi Algoritium, 2.3. Shortest Path Algorithms 23,1. Label Correcting Methods 2.3.2. Label Correcting Variations - A* Algorithm 23.8, Brancand-Bound . 234. Constrained and Multiobjective Problems 2. Notes, Sourees, and Exercises 3. Deterministic Contin \ous-Time Optimal Control 3.1. Continuous-Time Optimal Control 32, ‘The Hamilton-Jacobi-Bellinan Bquation 3.3. ‘The Pontryagin Minimum Principle 3.3.1. An Informal Derivation Using the HJB Equation 3.3.2. A Derivation Based on Variational Ideas 3.3.3. Minitmum Principle for Discrete-Time Problems 3.4. Extensions of the Minimuin Principle 3.4.1. Fixed Terminal State 3.4.2, Pree Initial State 2 plz pei8 p. 35 pai p. 46 pS p64 p. 68 =p. 70 pT p78 p87 p. BS: pO p. 97 p. 106 p. 109) pis pls p. 125 p. 129 p13 p.1SL p. 135Contents 34.8, Pree Terminal Time 135; 344, Time-Varying Systom and Cost p. 138 8.4.5, Singular Problems p. 139 85, Notes, Sources, and Exercises p. M2 Problems with Perfect State Information 4.1. Linear Systems and Quadratic Cast p. ds 4.2 Inventory Control p. 162 43. Dynamic Portfolio Analysis p.170 44. Optimal Stopping Problems». | p.176 4.5, Scheduling and the Interchange Argument . 186 4.6. Set-Membership Description of Uncertainty . 190 46.1. Set-Membership Estimation... 2 pil 46.2. Control with Unknown-but-Bounded Disturbances p. 197 47. Notes, Sources, and Exercises 201 + Problems with Imperfect State Information 5:1. Reduction to the Perfect Information Case p.218 5.2 Linear Systems and Quadratie Cost p. 229 5.8, Minimum Variance Control of Linear Systems | 236 54, Sulicent Statistics and Finite-State Markov Chains | |p. 251 5.4.1, The Conulitional State Distribution p. 252 5.42. PiniteState Systems 258; 5.5, Notes, Sourees, and Exercises p.270 Suboptimal Controt 6.1. Certainty Equivalent and Adaptive Control . 283 6.1.1. Caution, Probing, and Dual Control p. 280 6.1.2, Two-Phase Control and Identifiablity | | p. 291 6.1.3, Certainty Bquivalent Control and Identifiability |p. 208 6.14. Sel-Tyming Regulators ; p. 298 Open-Loop Feedback Control . 300 Limited Lookabead Policies. 2 2 | | . 304 6.3.1 Performance Bounds for Limited Lookahead Policies p. 308, 63.2. Computational Issues in Limited Lookahead . . p. 310, 63.8. Problem Approximation - Enforced Decomposition p. 312 5.3, Aggregation : p. 319 63.5, Parametric Cost-to-Go Approximation . 835 6.4. Rollout Algorithms . : p. 335 6.4.1, Discrete Deterministic Problems p. 382 6.42. Q-Fuetors Evaluated by Simulation p. 361 64.3, Q-Pactor Approximation . 363 i | i / | i | | | i i | ! Contents 6.5, Model Predictive Control and Related Methods 6.5.1. Rolling Horizon Approximations 6.5.2. Stability Issues in Model Predictive Control 6.5.3. Restricted Structure Policies jonal ‘Topics in Approximate DP tization ‘i 6.6.2. Other Approximation Approaches 6.7. Notes, Sources, and Exercises 66. 7, Introduction to Infinite Horizon Problems TAL. An Overview 7 7.2. Stochastic Shortest Path Problems 7.3. Discounted Problems 7A. Average Cost per Stage Problems 7.5. Semi-Markov Problems 7.6. Notes, Sonroes, and Exercises Ap yendix A: Mathematical Review AL. Sets A.2. Buclidean Space . A. Matrices AA. Analysis A.5. Convex Sets and Pinetions Appendix B: On Optimization Theory B.1. Optimal Solution B.2. Optimality Conditions. B.3. Minimization of Quadratic Forms Appendix C: On Probal G.1. Probability Spaces C22. Random Variables C.8. Conditional Probability ity Theory Appendix D: On Finite-State Markov Chains Da. Stationary Markov Chains D2. Classification of States D.3. Limiting Probabilities Dal, First Passage Times p- 402 p. 405, p.4l7 pa p. 435, p45, p59) p. 460 p46 p. 465, p. 467 p. 468: p. 470 p47 p. 472 para p. 475 parr p. ATS p. 479 p. 480Contents Appendix E: Kalman Filtering F.1, Least-Squares Bstimation pdBL B.2, Linear Least-Squares Estimation. p- 483. 18. State Estimation ~ Kalinan Filter. p49 EAA, Stability Aspects + p96 E.5, Gauss-Markov Estimators no) B.6, Deterministic Least-Squares Estimation p. 501 Appendix F: Modeling of Stochastic Linear Systems P.1, Linear Systems with Stochastic Inputs ~ p. 503 F.2, Processes with Rational Spectrum 7 p- 904 FB. The ARMAX Model. 506 Appendix G: Formulating Problems of Decision Under Uncer- tainty G.1, The Problem of Decision Under Uncertainty p. 507 G2, Expected Utility Theory and Risk . : + pall GB. Stochastic Optimal Control Problems p. 524 References p. 529 Index p. Sal Contents CONTENTS OF VOLUME It 1. Inflnite Horizon ~ Discounted Problems 1.1, Minimization of Total Cost ~ Introduetion 1.2, Discounted Problems with Bounded Cost per Stage 1.3. Finite-State Systems ~ Computational Methods 1.3.1. Value Iteration and Error Bounds 1.32, Policy Iteration 1.3.3. Adaptive Aggregation 1.34, Linear Programming 1.3.5, Limited Lookahead Policies 1.4, ‘The Role of Contraction Mappings 1.5. Scheduling ane Multiarmed Bandit, Problems 1.6. Notes, Sourees, and Exercises 2, Stochastic Shortest Path Problems 2.1, Main Results 2.2, Computational Methods 2.2.1. Value Iteration 2.2.2. Policy Hleration 2.3, Simnlation-Based Methods 2.3.1. Policy Evaluation by Monte-Carlo Simulation 28.2. Q-Learning 2.3.8. Approximations 2.3.4, Extensions to Discounted Problems 23.5. The Role of Parallel Computation 2A, Notes, Sourees, and Exercises 3. Undiscounted Problems 3.1. Unbounded Costs per Stage 3.2. Linear Systems and Quadratic Cost, 3.3. Inventory Control 34. Optimal Stopping 3.5, Optimal Gambling Strategies 3.6. Nonstationary and Periodic Problems 8.1. Notes, Sources, and Bxercises 4. Average Cost per Stage Problems 4.1, Preliminary Analysis 4.2. Optimality Conditions 43. Computational Methods 43.1. Value Iterationx Contents 4.3.2. Policy eration 43.3. Linear Programming 4.3.4 Simulation-Based Methods 44, Infinite State Space 4.5. Notes, Sources, and Exercises 5. Continuous-Time Problems 5.1. Uniformization 2, Queueing Applications 5.3. Semi-Markov Problems 5.4, Notes, Sources, and Exercises References Index Preface ‘This two-volume book is based on a first-year graduate couse on dynamic programming and optimal control that I have taught for over twenty years at Stanford University, the University of Illinois, and te Mas- sachusetts Institute of Technology. ‘The course has been typically attended by students from engineering, operations research, economies, and applied. ‘mathematics. Accordingly, a principal objective of the hook has been to provide a unified treatment of the subject, suitable for a broad audience, In particular, problems with a continuous character, such as stochastic com. ‘rol problems, popular in modern control theory, ate sinnltaneously treated with problems with a discrete character, such as Markovian decision prob- Jems, popular in operations research. Furthermore, many applications ane examples, drawn from a broad variety of fields, are discussed ‘The book may be viewed as a greatly expanded and peclagogically improved version of my 1987 book “Dynamic Programming: Deterministic and Stochastic Models,” published by Prentice-Hall. I have included much new material on deterministic and stochastic shortest path problems, as well as a new chapter on continuous-time optimal control problems and the Pontryagin Minium Principle, developed from a dynamic programraing viewpoint, T have also added a fairly extensive exposition of simulation. based approximation techniques for dynamic programming. ‘These tech- iques, which are often referred to ns “nenro-dynamic programming” or “reinforcement learning,” represent a breakthrough in the practical ap- lication of dynamic programming to complex problems that involve the ‘dual curse of large dimension and lack of an accurate mathematical model. Other material was also augmented, substantially modified, and updated With the new material, however, the book grew so mnch in size that it became necessary to divide it into two volumes: one on finite horizon, and the other on infinite horizon problems. ‘This division was not only natnral in terms of size, but also in terms of style and orientation. ‘The first volume is more oriented towards modeling, and the second is more ‘oriented towards mathematical analysis and compntation. T have included in the first volume a final chapter that provides an introductory treatme of infinite horizon problems. ‘The purpose is to make the first volume sexi Preface contained for instructors who wish to cover a modest, amount. of infinite horizon material in a contee that is primarily oriented towards modeling, conceptualization, and finite horizon problems, ‘Many topies in the book are relatively independent of the others. For exainple Chapter 2 of Vol. I on shortest path problems can be skipped without loss of continuity, and the same is trae for Chapter 3 of Vol. I, which deals with continuous-time optimal control. As a result, the book can be used to teach several different types of courses. (a) A bwo-semester course that covers both volumes. (b) A one-semester course primarily focnsed on finite horizon problems that covers: most of the first. volume, (6) A one-semester course focused on stochastic optimal control that cov- ‘ers Chapters 1, 4, 5, and 6 of Vol. I, and Chapters 1, 2, and 4 of Vol. n (a) A one-semester course that covers Chapter 1, about 50% of Chapters 2 through 6 of Vol. I, and about 70% of Chapters 1, 2, and 4 of Vol. 1. ‘This is the course I usually teach at MIT: (e) A one-quarter engineering course that covers the first three chapters ‘and parts of Chapters 4 trough 6 of Vol. 1 (f) A one-quarter mathematically oriented course focused on infinite hori ‘zon problems that covers Vol. I ‘The mathematical prerequisite for the text is knowledge of advanced calculus, introductory probability theory, and matrix-vector algebra. A summary of this material is provided in the appendixes. Naturally, prior exposure to dynamic system theory, control, optimization, or operations research will be helpful to the reader, but based on my experience, the material given here is reasonably self-contained. he book contains a large number of exercises, and the serious render will benefit greatly by going through them, Solutions to all exercises are comnpiled in a mantial that is available to instructors from the author. Many thanks are due to the several people who spent long hours contributing ( this manual, partienlarly Steven Shreve, Erie Lotederman, Lakis Poly- tmenakos, and Cynara Wa, Dynatie programming is a conceptually simple technique that can bbe adequately explained using elementary analysis. Yet a mathematically vigorous treatment of general dynamic programming requires the compl cated machinery of measure-theoretic probability. My choice has been to bypass the complicated mathematies by developing the subject in general- ity, while claiming rigor only when the underlying probability spaces are countable, A mathematically rigorous treatment of the subject is carried cont in my monograph “Stochastic Optimal Control: ‘The Discrete Time | Preface val Case,” Academie Press, 1978, coauthored by Steven Shreve. ‘This mono- graph complements the present text and provides a solid foundation for the subjects developed somewhat informally here. Finally, Tam thankful to a number of individuals and institutions for their contributions to the book. My understanding of the subject was sharpened while I worked with Steven Shreve on our 1978 monograph. ‘My interaction and collaboration with John Tsitsiklis on stochastic short- cst paths and approximate dynamic programming, have been most. valu- able. Michael Caramanis, Bmmanuel Fernande-Caucherand, Pierre Hus- blet, Lennart Ljung, and John Tsitsiklis tanght from versions of the book, and contributed several substantive comments and homework problems. A number of colleagues offered valuable insights and information, particularly David Castanon, Eugene Feinberg, and Krishna Pattipati. NSF provided research support. Prentice-Hall graciously allowed the use of material from my 1987 book, ‘Teaching and interacting with the students at MIT have kept up my interest and excitement for the subject. Dimitsi P. Bertsekas Spring, 1995 {Note added in the Srd edition: ‘This monograph was republished by Athena Scientific in 19916, and can also be freely downloaded from the anthor's www site: ‘tip web anit.edu/dimiteb ww home. htm.xiv Preface Preface to the Second Edition ‘This second edition has expanded by nearly 30% the coverage of the origi- tual. Most of te new material is concentrated in four areas: (9) In Chapter 4, section was added on estimation and control of sys- toms with a non-probabilistic (set membership) description of uncer- luinty. ‘This subject, a personal favorite of the author since it was the subject of his 1971 Ph.D. thesis, has become popular, as minimax and Hye control methods have gained inereased prominence. (b) Chapter 6 was doubled in size, to reflect the popularity of subopti- ‘mal control and neuro-dynamic programming methods. In particular, the coverage of certainty equivalent, and limited lookahead methods hhas been substantially expanded. Furthermore, a new section was added on neuro-dynamie programming and rollout algorithms, and their applications in combinatorial optimization and stochastic opti ‘mal control (©) In Chapter 7, an introduction to continnous-time, semi-Markov deci- sion problems was added in a separate last section, (a) A new appendix was included, which deals with various formulations of problems of decision under uncertainty. ‘The foundations of the ‘minimax and expected utility approaches are framed within a broader context, and some of the aspects of utility theory are discussed. ‘There are also miscellaneous additions and improvements scattered through out the text, and a more detailed coverage of deterministic problems is given in Chapter 1. Finally, a new internet-based feature was added to the book, which extends its scope and coverage. Many of the theoretical ‘exercises have been solved in detail and their solutions have been posted in the book's www page http: //www.athenase.com/aphook-itml pk ‘These exercises have been marked with the symbol @) | would like to express my thanks to the many colleagues who con- tributed snggestions for improvement of the second edition. Dimitai P. Bertsekns all, 2000, | Preface aw Preface to the Third Edition ‘The third edition contains a substantial amount of new material, particu larly on approximate dynamic programming, which has now become one of the principal focal points of the book, In parti (a) The subject of mini eluding a new section in Chapter 6, 1ax control was developed in greater detail, in Chapter 1, which connects with new material (b) The section on anction algorithms for shortest paths in Chapter 2-was eliminated. ‘These methods are not currently used in dynamic pro- gramming, and a detailed discussion has been provided in a chapter from the author's Network Optimization book. ‘This chapter ean be freely downloaded from ttps/ /web.mit.edu/dimiteib /wow /net html (e) A section was added in Chapter 2 on dynamic programming and shortest path algorithms for constrained and multiobjective problems, (@) ‘The material on sufficient statisties and partially observable Markow decision problens in Section 5.4 was resteuetured ancl expanded. was added in Chapter 6: (2) An expanded discussion of one-step lookahead policies andl as sociated performance bounds in Section 6.3.1. (e) Considerable new mater (2) A discussion of aggregation methods and diseretization of conti- uous-state problems (see Subsection 6.34). (8) A discussion of model predictive control and its relation to other suboptimal control methods (see Subsection 6.5.2) (4) An expanded treatment of open-loop feedback control and re- lated methods based on a restricted structure (see Subsection 65.3), Thave also added a few exercises, and revised a few sections while preserving their essential content. ‘Thanks are due to Haixia Lin, who worked out several exercises, and to Janey Yu, who reviewed some of the new sections and gave me valuable feedback. Dimitri P. Bertsekas |hetps/ /web.anit.edu/dimitrib /www /home.btml Summiner 2005The Dynamic Programming Algorithm Contents 1.1. Introduction 1.2. The Basic Problem 1.3. The Dynamic Programming Algorithm 1.4. State Augmentation and Other Reformulations 1.5. Some Mathematical Issues 1.6. Dynamic Programming and Minimax Control LT. Notes, Sources, and Exercises .aa ‘The Dynamic Programming Algorithm Chap. 1 Life can only be understood going backwards, but it must be lived going forwards. Kierkegaard INTRODUCTION ‘This book deals with situations where decisions are made in stages. ‘The outcome of each decision may not be fully predictable but can be antici- ted to some extent before the next decision is made. The objective is to Ininimize a certain cost ~ a mathematical expression of what is considered san undesirable onteome, A key aspect of such situations is that decisions cannot be viewed in isolation since one must balance the desire for low present cost with the uundesirability of high future costs. "The dynamie programming technique eaplures this tradeoff. At each stage, it ranks decisions based on the sin of the present cost and the expected future cost, assuming optimal decision ‘making for subsequent stages, ‘There isa very brond variety of practical problems that can be treated by dynamic programming. In this book, we try to keep the main ideas ‘uncluttered by irrelevant assumptions on problem structure, To this end, wwe formulate in this section a broadly applicable model of optimal control of a dynamic system over a finite number of stages (a finite horizon). ‘This ‘model will occupy us for the first six chapters; its infinite horizon version will be the subject of the last chapter as well as Vol. I. ‘Our basic model has two principal features: (1) an underlying diserete- tine dynamic system, and (2) a cost function that is additive over time. ‘The dynamic system expresses the evolution of some variables, the systern’s “stato”, under the influence of decisions made at discrete instances of time. ‘The system has the form Hep = fultey te, we), B= O,1,...,N— where ik indexes discrete time, v4 is the state of the system and summatizes past information that is relevant for fature optimization, ‘ug is Uke contro! or decision variable to be selected at time ky ‘wh is a random parameter (also called disturbance or noise depending on the context), eT | | } i | | | i ‘See. 1.1 Introduction N jis the horizon or nnmber of ti nes control is applied, and fe i function that describes the system and in pasticular the mech- anism by which the state is updated ‘The cost fumetion is nitive in the sense that the cost incurred at time k, denoted by g.(z%,ux, we), accumulates over time, The total cost swan) + oul, 1 2), fod where gy(zw) is a terminal cost incurred at the end of the process. How- ever, because of the presence of wy, the cot is generally a random vasiable and cannot be meaningflly optimized. We therefore forte the problem aS an optimization of the expected cost B{ovin) ' Satomi}, where the expectation is with respect to the joint distribution of the random variables involved. ‘The optimization is over the controls wo, t,.-.%4n 1, but some qualification is needed here; each control 1, is selocted with some knowledge of the current state ax (either ils exact value or some other related information) ‘A more precise definition of the terminology just used will be given shortly. We first provide some orientation by means of examples, Example 1.1.1 (Inventory Control) Consider a problem of ordering @ quantity of a certain item at each of NV periods s0 as to (roughly) moot a stochastic dematxl, while minimizing the incurred expected eost. Let us denote {Fe stork available at the beginning of the kth period, ‘uy stock ordered (and iminediately delivered) at the beginning of the kth peti, ‘wy demand during the kth period with given probability distribution ‘We assume that wp,ws,...,20-1 are independent random variables, fand that excess demand is backlogged and filled as soon a additional inven tory becomes available. ‘Thus, stock evolves according: to the discrete-time ‘equation Hiya = ah tun — wa, \here negative stock corresponds to backlogged demand (se Fig, 11.1), ‘The cost incurred in period & consists of two component: (8) A cost (zy) representing a penalty for ether positive stock x. (holding cost for excess inventory) or negative stock 2x (shortage cost for unfilled demand)‘The Dynamic Programming Algorithm Chap. 1 vi | Demand at Peo ‘Stock at Patio’ k m% Inventory System Stock at Poiod +1 Hee = We Stock ordered at Periodic Cost of Period k 4 On) + eu Figuro Ld Inventory contrat example. At petiod hy the current stack (Gtate) 24, the stack ordered (control) ty, and the dematd (random distur bance) wy determine the cost r(zq) Fev and the tock 1 — 24 hve —Wh at the next period (b) ‘The purchasing cost cus, where ¢ is east per unit ordered, ‘There is also a terminal cost (ew) for being left with inventory zy at the fend of IV periods. Thus, the total east over NV periods is »f amr $2 e4e-rou)} ee re ee eal (eee ee ee Pagaetegr pera prementierreretriegoor Pe rn hare eteviep a tae cea ee cen eee Sane oe ee ne ce: Geta ee Geta ee a eee a nee, ee ee ee fu cece haeeaian tee vase eh ees earns te Se ne eee eee fe eran oleae ee ee ne ene eee: Seats ea cen are geet Berdebrs regep rector geclerepernt cpr en ee ee See. 11 Introduction 50 as to minimize the expected cost. ‘The meaning of jy i ‘and each possible value of 2, hat, for each u(x) = amonnt that should be ordered at kine kif the stock is ‘The sequence x = {j.--.44n-— control low, For each x, the correspond will be referred to as a policy or cost for fixed initial stock is dele) = »{ nem +See rants}, fand we want to minimize Je(1o) for a given xy over all sr that satisfy the constraints of the problem. ‘This isa typical dynamic programming problew We will analyze this problem in various forms in subsequent sections. For example, we will show in Section 4.2 that for a reasonable choice of the cost function, the optimal ordering policy is oft form sales) = {54 aa < Su 0 otherwise, where Si is suitable threshold level determined by the data of the problem, In other words, when stock falls below the threshold Si, order just enough to bring stock up to 5. ‘The preceding example illustrates the main ingredients of the basic problem formulation: (a) A discrete-time system of the form usr = feleasun, we), where fi, is some function; for example in the inventory ease, we have Filth, the, We) = te + ue — we. (b) Independent random parameters wy. ‘This will be generalized by al- lowing the probability distribution of wy, to depend on sry and uy; in the context of the inventory example, we can think of a situation where the level of demand wy is influenced by the eurrent stock level (c) A control constraint; in the example, we have u, > 0. In general, ‘the constraint set will depend on 2. and the time index k, that uy € Ug(). ‘To see how constraints dependent on xy can arise in the inventory context, think of a situation where there is an upper bound Bon the level of stock that can be accommodated, $0 uy < By (a) Am additive cost of the form #{ ante + aeons ta, oy},‘The Dynamic Programming Algorithin Chap. 1 where gy are some funetions; in the inventory exam aren) = Ran), gues este) = r(ra) + eum (©) Optimization over (closed-loop) policies, that is, rules for choosing 1 for each k and each possible value of ay. Discrete-State and Finite-State Problems 1n the preceding example, the state 2x was a continuous real variable, and it is easy to think of innltidimensional generalizations where the state 1s ‘an n-dimensional vector of real variables. It i also possible, however, that the state takes values from a diserete set, such as the integers ‘A version of the inventory problem where a discrete viewpoint is more natural arses when stock is measured in whole units (such as cars), each of which is significant fraction of 2p, up, oF my. Te is mote appropriate ten to take as state space the set ofall integers rather than the set of real umbers. ‘The form of the system equation and the cost per period will, of course, stay the sume, Generally, there are many situations where the state is naturally dis crete and there is no continnous counterpart of the problem. Such sit uations are often conveniently specified in terms of the probabilities of transition between the states. What we need to know is piu, which is the probability at time & that the next slate will be J, given that the ‘current state is i, and the control selected i tie vist k) = Press = 3 |e = iu =u), ‘This type of state transition can alternatively be described in terms of the discrete-time system equation where the probability distribution of the random parameter wn is Pan = 5) H} = Pils k) Conversely, given a discrete-state system in the form au flrs ons te), together with the probability distribution P,(we | 24, uz) of wy, we can provide an equivalent transition probability description. ‘The corresponding ‘transition probabilities are given by ists k) = Pe{ Welds 3) |e See. 11 Introduction : whore W(,1,3) isthe set Webi) {w | i= felis ww). ‘Thus a discrete-state system can equivalently be described in terms of a difference equation or in terms of transition probabilities, Depend ing on the given problem, it may be notationally or mathematically more ‘convenient to use one description over the other, ‘The following examples illustrate diserete-state problems. ‘The first example involves a deterministic problem, that is, a problem where there jy no stochastic uncertainty. In such problem, wh aba given state, the next state is fully determined; that is, for any state i, control u, and time k, the transition probability p,;(u, ) is equal to 1 for & single state j, and itis 0 for all other candidate next states. The other three examples involve stochastic problems, where the next state resulting from a given choice of control at a given state cannot be determined a priori n a control is chosen ‘Example 1.1.2 (A Deterministic Scheduling Problem) Suppose that to produce a certain product, fonr operations must be perforned fon a certain machine. ‘The operations are denoted by A, B, C, and D. We assume that operation B can be performed only after operation A. has been performed, and operation D can be performed only after operation Ty has heen performed. (‘Thus the sequence CDAB ig allowable but the sequence CDBA is not.) ‘The setup cast Chan for passing from any operation m Lo any other operation 2 is given. ‘There is also an initial startup east $4 or Se for starting with operation A or C, respectively. ‘The cost of a sequence is the ‘sum of the seLup costs associated with i; for example, the operation sequence ACDB has cost Sat Cao + Cov + Con ‘We can view this problem as a sequence of three decisions, naanely the choice of the first three operations to be perlormed (the last operation is determined from the preceding three), It is appropriate to consider as state te set of operations already performed, the initial state being an artificial state corresponding to the begining of the deeision process, "The possible state transitions corresponding to the possible states and decisions for this problem is shown in Fig. 11.2. Here the problem is deterministic, Le, at f given state, each choice of control leads Lo a uniquely determined state For example, at state AC the decision to perform operation D leads to state ACD with certainty, and has onst Coo. Deterministic problems with a finite ‘numberof states can be conveniently represented in terns of transition graphs such as the one of Fig. 1.1.2. ‘The optimal solution corresponds to the path hat starts at the initial state and ends at some state at the terminal time fand has minima sum of arc costs plus the terminal enst. We will study systematically problems of this type in Chapter 2.