# Welcome back

## Find a book, put up your feet, stay awhile

Sign in with Facebook

Sorry, we are unable to log you in via Facebook at this time. Please try again later.

or

Welcome to Scribd, the world's digital library. Read, publish, and share books and documents. See more

Download

Standard view

Full view

of .

Look up keyword

Like this

Share on social networks

5Activity

×

0 of .

Results for: No results containing your search query

P. 1

RIchard Bellman on the Birth of Dynamic ProgrammingRatings: (0)|Views: 276|Likes: 4

Published by Jeff Pratt

See more

See less

https://www.scribd.com/doc/8185567/RIchard-Bellman-on-the-Birth-of-Dynamic-Programming

08/25/2010

text

original

RICHARD BELLMAN ON THE BIRTH OFDYNAMIC PROGRAMMING

STUART DREYFUS

University of California, Berkeley, IEOR, Berkeley, California 94720, dreyfus@ieor.berkeley.edu

W

hat follows concerns events from the summer of 1949, when Richard Bellman ﬁrst became inter-ested in multistage decision problems, until 1955. AlthoughBellman died on March 19, 1984, the story will be toldin his own words since he left behind an entertaining andinformative autobiography,

Eye of the Hurricane

(WorldScientiﬁc Publishing Company, Singapore, 1984), whosepublisher has generously approved extensive excerpting.During the summer of 1949 Bellman, a tenured asso-ciate professor of mathematics at Stanford University witha developing interest in analytic number theory, was con-sulting for the second summer at the RAND Corporationin Santa Monica. He had received his Ph.D. from Princetonin 1946 at the age of 25, despite various war-relatedactivities during World War II—including being assignedby the Army to the Manhattan Project in Los Alamos.He had already exhibited outstanding ability both in puremathematics and in solving applied problems arising fromthe physical world. Assured of a successful conventionalacademic career, Bellman, during the period under con-sideration, cast his lot instead with the kind of appliedmathematics later to be known as operations research. Inthose days applied practitioners were regarded as distinctlysecond-class citizens of the mathematical fraternity. Alwaysone to enjoy controversy, when invited to speak at vari-ous university mathematics department seminars, Bellmandelighted in justifying his choice of applied over pure math-ematics as being motivated by the real world’s greater chal-lenges and mathematical demands.Following are excerpts, taken chronologically fromRichard Bellman’s autobiography. The page numbers aregiven after each. The excerpt section titles are mine. Theseexcerpts are far more serious than most of the book, whichis full of entertaining anecdotes and outrageous behaviorsby an exceptionally

human

being.

Stuart Dreyfus

BELLMAN’S INTRODUCTION TO MULTISTAGEDECISION PROCESS PROBLEMS

“I was very eager to go to RAND in the summer of 1949

I became friendly with Ed Paxson and asked himwhat RAND was interested in. He suggested that I work on multistage decision processes. I started following thatsuggestion” (p. 157).

CHOICE OF THE NAME DYNAMICPROGRAMMING

“I spent the Fall quarter (of 1950) at RAND. My ﬁrst task was to ﬁnd a name for multistage decision processes.“An interesting question is, ‘Where did the name,dynamic programming, come from?’ The 1950s were notgood years for mathematical research. We had a very inter-esting gentleman in Washington named Wilson. He wasSecretary of Defense, and he actually had a pathologicalfear and hatred of the word, research. I’m not using theterm lightly; I’m using it precisely. His face would suffuse,he would turn red, and he would get violent if people usedthe term, research, in his presence. You can imagine how hefelt, then, about the term, mathematical. The RAND Cor-poration was employed by the Air Force, and the Air Forcehad Wilson as its boss, essentially. Hence, I felt I had to dosomething to shield Wilson and the Air Force from the factthat I was really doing mathematics inside the RAND Cor-poration. What title, what name, could I choose? In the ﬁrstplace I was interested in planning, in decision making, inthinking. But planning, is not a good word for various rea-sons. I decided therefore to use the word, ‘programming.’I wanted to get across the idea that this was dynamic, thiswas multistage, this was time-varying—I thought, let’s killtwo birds with one stone. Let’s take a word that has anabsolutely precise meaning, namely dynamic, in the clas-sical physical sense. It also has a very interesting propertyas an adjective, and that is it’s impossible to use the word,dynamic, in a pejorative sense. Try thinking of some com-bination that will possibly give it a pejorative meaning.It’s impossible. Thus, I thought dynamic programming wasa good name. It was something not even a Congressmancould object to. So I used it as an umbrella for my activi-ties” (p. 159).

EARLY ANALYTICAL RESULTS

“The summer of 1951 was old-home-week. Sam Karlin andHal Shapiro were at RAND.

Subject classiﬁcations:

Dynamic programming: history. Professional: comments on.

Area of review:

Anniversary Issue (Special).

Operations Research

© 2002 INFORMSVol. 50, No. 1, January–February 2002, pp. 48–51

48

0030-364X/02/5001-0048 $05.001526-5463 electronic ISSN

Dreyfus /

49“Hal Shapiro, Ted Harris, and I worked on an interest-ing functional equation which arose in learning processes.Although we wrote a long paper on this, we never pub-lished our results. Sam Karlin obtained some results con-cerning this equation which he published in the

Paciﬁc Journal of Mathematics

. I had studied a very interestingfunctional equation which arose in multi-strike analysis, thegold mining equation. The solution was given most easilynot in terms of the unknown function, but in terms of anaction or decision. This intrigued me, because I had neverseen this phenomenon before.“Hal Shapiro, Sam Karlin, and I tried very hard to solvethe general case. It was like a problem in number theory,the solution seemed close, but outside one’s grasp.“I did manage to handle a continuous version, whichshowed why the original problem was so difﬁcult.“Hal Shapiro and I took advantage of proximity to dosome work on number theory. We greatly enjoyed workingtogether again.“Meanwhile, of course, I kept up the investigation of dynamic programming processes” (p. 160).

THE MODERN MATHEMATICAL INTELLECTUAL

“I had to make a major decision. Should I return to Stanfordor stay at RAND? I had thought about this question inPrinceton, but it was not an easy decision to make sincethere were strong arguments on each side.“At Stanford, I had a tenured position, good for anotherthirty-eight years. The retirement age at Stanford was sev-enty. I also had a good teaching position, with not too muchteaching, and a ﬁne house, which I have described above.But these were not the important considerations. At Stan-ford I had a chance to do analytic number theory, which Ihad wanted to do since I was sixteen.“However, I had to face the fact that I could not do whatI wanted to do. Possibly the state of mathematics did notallow this. Certainly, my state of knowledge was not upto it.“I had spent enough time in Los Angeles to know that Iwould enjoy living there. I also knew that Los Angeles hadmany ﬁne houses, although it was not until 1968 that I hadone that was better than the one up in Stanford.“I was intrigued by dynamic programming. It was clearto me that there was a good deal of good analysis there.Furthermore, I could see many applications. It was a clearchoice. I could either be a traditional intellectual, or a mod-ern intellectual using the results of my research for theproblems of contemporary society. This was a dangerouspath. Either I could do too much research and too littleapplication, or too little research and too much applica-tion. I had conﬁdence that I could do this delicate activity,pie a la mode” (p. 173).

THE PRINCIPLE OF OPTIMALITY AND ITSASSOCIATED FUNCTIONAL EQUATIONS

“I decided to investigate three areas: dynamic program-ming, control theory, and time-lag processes.“My ﬁrst task in dynamic programming was to put it ona rigorous basis. I found that I was using the same tech-nique over and over again to derive a functional equation.I decided to call this technique “The principle of optimal-ity.” Oliver Gross said one day, ‘The principle is not rigor-ous.’ I replied, ‘Of course not. It’s not even precise.’ A goodprinciple should guide the intuition.“Some of the functional equations could be handled eas-ily using classical techniques. Some required a great dealof work. I wish I then had available the projected metric of Garrett Birkhoff. Many years later, Tom Brown and I wrotea short paper pointing out how useful this was.“Secondly, I turned to the study of the associated func-tional equations. I was not enthusiastic about doing this.The equations were highly nonlinear and unlike any othersthat had appeared in analysis. I was delighted when I foundthat a simple argument could handle most equations.“While doing this, I started work on control theory. I hadseen problems in economics and operations research and itwas clear that some effort was required. I had a good team,Irving Glicksberg and Oliver Gross. Both were talented andingenious mathematicians.“The tool we used was the calculus of variations. Whatwe found was that very simple problems required greatingenuity. A small change in the problem caused a greatchange in the solution.“Clearly, something was wrong. There was an obviouslack of balance. Reluctantly, I was forced to the conclusionthat the calculus of variations was not an effective tool forobtaining a solution” (pp. 174–175).

FORMULATION OF THE MARKOV DECISIONPROCESS PROBLEM

“I spent a great deal of time and effort on the functionalequations of dynamic programming. I was able to solvesome equations and to determine the properties of thefunction and the policy for others. I developed some newtheories, Markovian decision processes, and was able toreinterpret an old theory like the calculus of variations, of which I will speak more about below” (p. 178).

DYNAMIC PROGRAMMING AND OPTIMALCONTROL THEORY

“A number of mathematical models of dynamic program-ming type were analyzed using the calculus of variations.The treatment was not routine since we suffered either fromthe presence of constraints or from an excess of linearity.An interesting fact that emerged from this detailed scrutinywas that the way one utilized resources depended criticallyupon the level of these resources, and the time remainingin the process. Naturally this was surprising only to some-one unversed in economics such as myself. But this wasmy condition, with the result that the observation of thisphenomenon came as quite a shock. Again the intriguingthought: A solution is not merely a set of functions of time,

50 /

Dreyfus

or a set of numbers, but a rule telling the decisionmakerwhat to do; a policy.“The mathematical structure of these perplexing analyticproblems was quite open. What was remarkable was thelevel of analytic intricacy of solution introduced by simpleconstraints. These constraints were an essential part of thebackground, introduced by immediate economic, engineer-ing, and military considerations.“A problem of LaSalle’s which caught my attention at thesame time was the ‘bang-bang’ control problem. This was aquestion of restoring a system to equilibrium as quickly aspossible subject to constraints in the restoring force. Thiswas a problem closely related to stability theory.“As a result of a detailed study of dozens of variationalproblems of the foregoing type, and ﬁlling hundreds of pages with equations and calculations, it became quite clearthat there would never be any elegant, uniform way of solv-ing problems of this genre in analytic terms. Each individ-ual problem was an exercise in ingenuity, much like planegeometry. Change one small feature, and the structure of the solution was strongly altered. There was no stability!“Consequently, if one really wanted to obtain numeri-cal solutions to variational problems in an effective fash-ion, we needed some other tools. I was reluctant to becomeover-involved, since all along I had no desire to work seri-ously in the calculus of variations. A course in the sub- ject in college had given me simultaneously a rather lowopinion of its intrinsic interest and a healthy respect for itsintricacies. It appeared to be ﬁlled with complicated exis-tence and uniqueness theorems with self-imposed restric-tions, none pointing in any particular direction. This ispertinent to a comment made by Felix Klein, the greatGerman mathematician, concerning a certain type of math-ematician. When this individual discovers that he can jumpacross a stream, he returns to the other side, ties a chairto his leg, and sees if he can still jump across the stream.Some may enjoy this sport; others, like myself, may feelthat it is more fun to see if you can jump across biggerstreams, or at least different ones.“Despite my personal feelings, the challenge remained.How did one obtain the numerical solutions of optimiza-tion problems? Were there reliable methods? As pointedout above, I did not wish to grapple with this thorny ques-tion, and I had certainly not contemplated the applicationof dynamic programming to control processes of determin-istic types. Originally, I had developed the theory as a toolfor stochastic decision processes. However, the thought wasﬁnally forced upon me that the desired solution in a con-trol process was a policy: ‘Do thus-and-thus if you ﬁndyourself in this portion of state space with this amount of time left.’ Conversely, once it was realized that the conceptof policy was fundamental in control theory, the mathe-maticization of the basic engineering concept of ‘feedback control,’ then the emphasis upon a state variable formula-tion became natural. We see then a very interesting inter-action between dynamic programming and control theory.This reinforces the point that when working in the ﬁeld of analysis it is exceedingly helpful to have some underlyingphysical processes clearly in mind.“What is worth noting about the foregoing developmentis that I should have seen the application of dynamic pro-gramming to control theory several years before. I shouldhave, but I didn’t. It is very well to start a lecture by saying,‘Clearly, a control process can be regarded as a multistagedecision process in which

,’ but it is a bit misleading.Scientiﬁc developments can always be made logical andrational with sufﬁcient hindsight. It is amazing, however,how clouded the crystal ball looks beforehand. We all wearsuch intellectual blinders and make such inexplicable blun-ders that it is amazing that any progress is made at all.“All this contributes to the misleading nature of conven-tional history, whether it be analysis of a scientiﬁc discov-ery or of a political movement. We are always looking atthe situation from the wrong side, when events have alreadybeen frozen in time. Since we know what happened, it isnot too difﬁcult to present convincing arguments to justify aparticular course of events. None of these analyses must betaken too seriously, no more than Monday morning quar-terbacking.“I strongly recommend the interesting study of theseand related matters by Jacques Hadamard, the great Frenchmathematician, in his book

The Psychology of Inventionin the Mathematical Field

(Dover Publications, New York,1945: paperback)” (pp. 180–182).

A SYSTEMATIC METHODOLOGICALAPPROACH TO MATHEMATICS

“As pointed out above, as of 1954 or so I had stumbled intosome important types of problems and had been pushed,willy-nilly, into answering some signiﬁcant kinds of ques-tions. I could handle deterministic control processes tosome extent and stochastic decision process in economicsand operations research as well. Where next? At this point,I began to think in a logical fashion, using a systematicmethodological approach. The point about the suitable phi-losophy preparing one for the fortunate accident should bekept in mind.“There are several ways in which a mathematician canproceed to extend his research efforts, particularly one whois deeply interested in problems arising from the physicalworld. He can, on one hand, examine the equations he hasbeen working with and modify them in a variety of ways.Or he can ask questions that have not been asked beforeconcerning the nature of the solution of the original equa-tions. This is basically a very difﬁcult way to carry outresearch. It is very easy to change the form of an equa-tion in a large number of ways. The great majority of thenew equations are not meaningful, and, in consequence,lead to both difﬁcult and unimportant problems. Similarly,there are many questions that are difﬁcult to answer, buthardly worth asking. The well-trained mathematician doesnot measure the value of a problem solely by its intractabil-ity. The challenge is there, but even very small boys do notaccept all dares.

- Read and print without ads
- Download to keep your version
- Edit, email or read offline

© Copyright 2015 Scribd Inc.

Language

Choose the language in which you want to experience Scribd:

Sign in with Facebook

Sorry, we are unable to log you in via Facebook at this time. Please try again later.

or

Password Reset Email Sent

Join with Facebook

Sorry, we are unable to log you in via Facebook at this time. Please try again later.

or

By joining, you agree to our

read free for one month

Personalized recommendationsbased on books you love

Syncing across all your devices

Join with Facebook

or Join with EmailSorry, we are unable to log you in via Facebook at this time. Please try again later.

Already a member? Sign in.

By joining, you agree to our

to download

Personalized recommendationsbased on books you love

Syncing across all your devices

Continue with Facebook

Sign inJoin with emailSorry, we are unable to log you in via Facebook at this time. Please try again later.

By joining, you agree to our

Are you sure?

This action might not be possible to undo. Are you sure you want to continue?

CANCEL

OK

You've been reading!

NO, THANKS

OK

scribd