This action might not be possible to undo. Are you sure you want to continue?

BooksAudiobooksComicsSheet Music### Categories

### Categories

Scribd Selects Books

Hand-picked favorites from

our editors

our editors

Scribd Selects Audiobooks

Hand-picked favorites from

our editors

our editors

Scribd Selects Comics

Hand-picked favorites from

our editors

our editors

Scribd Selects Sheet Music

Hand-picked favorites from

our editors

our editors

Top Books

What's trending, bestsellers,

award-winners & more

award-winners & more

Top Audiobooks

What's trending, bestsellers,

award-winners & more

award-winners & more

Top Comics

What's trending, bestsellers,

award-winners & more

award-winners & more

Top Sheet Music

What's trending, bestsellers,

award-winners & more

award-winners & more

P. 1

Lagrange Multipliers|Views: 11|Likes: 0

Published by Priyanshu Bhattacharya

See more

See less

https://www.scribd.com/doc/94517483/Lagrange-Multipliers

05/23/2012

text

original

Dan Klein

1 Introduction

This tutorial assumes that you want to know what Lagrange multipliers are, but are more interested in getting the intuitions and central ideas. It contains nothing which would qualify as a formal proof, but the key ideas need to read or reconstruct the relevant formal results are provided. If you don’t understand Lagrange multipliers, that’s ﬁne. If you don’t understand vector calculus at all, in particular gradients of functions and surface normal vectors, the majority of the tutorial is likely to be somewhat unpleasant. Understanding about vector spaces, spanned subspaces, and linear combinations is a bonus (a few sections will be somewhat mysterious if these concepts are unclear). Lagrange multipliers are a mathematical tool for constrained optimization of differentiable functions. In the that we want to basic, unconstrained version, we have some (differentiable) function maximize (or minimize). We can do this by ﬁrst ﬁnd extreme points of , which are points where the gradient is zero, or, equivlantly, each of the partial derivatives is zero. If we’re lucky, points like this that we ﬁnd will turn out to be (local) maxima, but they can also be minima or saddle points. We can tell the different cases apart by a variety of means, including checking properties of the second derivatives or simple inspecting the function values. Hopefully this is all familiar from calculus, though maybe it’s more concretely clear when dealing with functions of just one variable. All kinds of practical problems can crop up in unconstrained optimization, which we won’t worry about here. One is that and its derivative can be expensive to compute, causing people to worry about how many evaluations are needed to ﬁnd a maximum. A second problem is that there can be (inﬁnitely) many local maxima which are not global maxima, causing people to despair. We’re going to ignore these issues, which are as big or bigger problems for the constrained case. In constrained optimization, we have the same function to maximize as before. However, we also have some restrictions on which points in we are interested in. The points which satisfy our constraints are refered to as the feasible region. A simple constraint on the feasible region is to add boundaries, such as insisting that each be positive. Boundaries complicate matters because extreme points on the boundaries will not, in general, meet the zero-derivative criterion, and so must be searched for in other ways. You probably had to deal with boundaries in calculus class. Boundaries correspond to inequality constraints, which we will say relatively little about in this tutorial. Lagrange multipliers can help deal with both equality constraints and inequality constraints. For the majority of the tutorial, we will be concerned only with equality constraints, which restrict the feasible region to points lying on some surface inside . Each constraint will be given by a function , and we will only be interested in points where .1

G @ A C 8 65 HFED97(4

A

A @ 8 65 B97(4

1 If

you want a

constraint, you can just move the to the left:

" ¨¦¤¢ £© © © § ¥ £¡ !

.

£©©©§ ¥ £ %%)(¦'¡

&

& 2301'¡ £

£

$ %£

#

This is the downward cupping paraboloid shown in ﬁgure 5. 2. 2. R %£ while satisfying R R %£ 2 0 £¡ Y¦¤¢ £ 0 0 R £ S vP X R r R £ £ StP X X & 2u0 £ 0 § £ R rsR qc f'¡ X 2 p0 R X 0W (i£ P I R £ P hI X 0 § £¡ g f'¢ 2 U0 S TP £ (1) (2) .5 1 1. while the unconstrained minimum is not even deﬁned (you can ﬁnd points with as low as you like). Here we can just substitute our value for (1) into .3 2.5 2 4 2 0 −2 −4 −6 −8 −10 2 1 0 0 −1 −2 −2 −1 1 2 2 Trial by Example Let’s do some example maximizations. However. with a constraint.5 0 −0. we’ll have an example of not using Lagrange multipliers. it highlights a basic way that we might go about dealing with constraints: substitution..5 2 1. Maximize the value of .5 −1 −0.5 1 0. P QI Figure 1: A one-dimensional domain. but we’ll come back to it once the Lagrange multipliers show up. First. Then.2 Substitution Let .5 −1 −1. To do this.1 A No-Brainer Let’s say you want to know the maximum value of subject to the constraint (see ﬁgure 1). and get our maximum value of . The unconstrained maximum is clearly at . we add the constraint .5 −2 −2 −1. we maximize (or minimize) by ﬁrst solving for one of the variables explicitly: S S P dcI 0 S eR aP 2 b0 £ XI ¨WP R £ R X P wgS X P ` I P VI Figure 2: The paraboloid . It isn’t the most challenging example..5 0 0. Now let’s say we constrain and to lie on the unit circle.

Left is the unit circle 0 R R r %£ R R %£ . which has a maximum at . dropping. let’s stick with the from the last section and consider how the Lagrange multiplier method would work. . the ellipse is tangent to the circle. while costs twice as much from . Beyond this value. At ﬁrst. it’s usually very hard to do. Since the contour (light curve) is a level curve. or level curves. Consider what happens as the ellipse expands. As the ballon expands. Since we may move anywhere along and still satisfy the constraint. the level curves do not intersect the circle. along with two different constraints. Since the curves aren’t tangent.5 −1 −0. the values of are high. The size-zero ellipse has the highest value of . we can nudge along to either side of the contour to either increase or decrease . S 0 X r £ 2 30 S 0 H & S S 0 a0 X . The minimum value occurs at both these boundary points. Again.5 −0. which are wide in the dimension. Note that the direction of greatest increase is always perpendicular to the level curves.5 1 1 0. The key thing is that. the two curves are tangent. and which represent points which have the same value of . are ellipses. So cannot be an extreme point.5 −1 −0. until the ellipse surrounds the circle and only the short axis endpoints are still touching. Imagine the ellipses as snapshots of an inﬂating and balloon.5 1 1.3 Inﬂating Balloons The main problem with substitution is that. 2 Differentiable curves which touch but do not cross are tangent.5 0 0. The contours.5 2 −2 −2 −1. as in ﬁgure 4(left). and no point on the level curve will be in the feasible circle. The dark circle in the middle is the feasible region satisfying the constraint . the value of along the ellipse decreases.5 0 0. despite our stunning success in the last section. then the curves will cross. This is the maximum constrained value for – any larger. ). This shouldn’t be too surprising.5 −1. where and . as in ﬁgrefﬁg:crossing(right). 2. If the two curves were not tangent.5 −2 −2 −1.5 −1 −1 −1. intersecting the circle at four points. Rather than inventing a new problem and discovering this the hard way. we’re stuck on a circle which trades for linearly. while the points on the other side have lower value.5 1. imagine a point (call it ) where they touch.5 1 1. we’re back to a one-dimensional unconstrained problem. the points to one side of the contour have greater value.2 2 1. meeting at .5 0. the one-dimensional problem is still actually somewhat constrained in that must be in . where and . but feel free to verify it by checking derivatives! RS S X 2 0 U£ T 2 0 0 aB£ S P S h § X X R X 2 30 & X R %£ 2 0 tB £ R XI (P S 2 0 3"£ 0 2 B § R R X ¨I X P wgS R XwsS VI P P ¡ P VI r R £ S ¡ 0 0 X 0 f'¢ § £¡ XI ¨yP & P xI Figure 3: The paraboloid right is the line .2 The ellipse then continues to grow. Figure 3(left) shows a contour plot of . When the long axis of the ellipse ﬁnally touches the circle at .5 0 0 −0. but the ellipse does not intersect the feasible circle anywhere. and highlights one weakness of this approach. The curves being tangent at the minimum and maximum should make intuitive sense. This is the minimum ( . Finding the constrained minimum here is slightly more complex.5 2 (3) and substitute into (4) (5) (6) Then. at . The arrows point in the directions of greatest increase of .

5 0. and nudge along that direction.96 −0.75 0 0 0. while the partial derivative with respect to recovers the constraint . . way of looking at the tangent requirement.5 0 0.1 −1. This should also make sense – the gradient is the direction of steepest ascent.04 −1. the gradient of will be perpendicular to as well. Consider again the zooms in ﬁgure 4.08 −0. but at least it illustrates the method. The contour is a level curve. So here’s another. equivalent.7 −0.0.8 0. It was so easy to solve with substition that the Lagrange multiplier method isn’t any easier (if fact it’s harder). Let’s re-solve the circle-paraboloid problem from above using this method. there is no local move you can make to increase which does not take you out of the feasible region .5 0. The is our ﬁrst Lagrange multiplier.04 0.08 1 0.94 −0. If the direction of steepest increase and decrease take you off perpendicularly off of . really!) rests on it. Formally.5 1 1. we can write our claim that the normal vectors are parallel at an extreme point as: So.02 −1 −0. The two curves being tangent at a point is equivalent to the normal vectors being parallel at that point. # & & STP I P P VI £ ¡ R RR £ r ¥R ¤ F£ R¥ £ & P £ '¡ '¢ £¡ 2 0 £ 3'¡ & & P £ ¦¤¡ ¦¤¢ £¡ & £ ¦¤¡ & ¡ # # 2 & & P # 0 § £ ¤¡ # £¡ ¤¢ # 2 # 0 ¢ # ¡ & 0'¡ £ 0 £¡ '¢ # 0 § £ %f'¡ & 2 0 £ 31'¡ 0 § £ f'¡ # 0 0 § £ ¤¡ £ & & & & (7) (8) (9) (10) (11) (12) (13) (14) (15) . then.8 −0. Now think about the normal vectors of the contour and constraint curves.06 −0.1 1. maxima.92 −1. which generalizes better.02 −0.6 −0.06 0.65 −1 −0. and. But that means that at an extreme point . the direction of steepest ascent had better be perpendicular to . while it is ﬁne for to have a non-zero gradient.04 0. get a non-zero direction along .6 Figure 4: Level curves of the paraboloid. is normal to it.7 −0. our method for ﬁnding extreme points3 which satisfy the constraints is to look for point where the following equations hold true: We can compactly represent both equations at once by writing the Lagrangian: and asking for points where The partial derivatives with respect to recover the parallel-normals equations. even if you are not at an unconstrained maximum of .5 −1. Otherwise. This intuition is very important. or neither. At a solution .02 0. The Lagrangian is: and we want 3 We can sort out after we ﬁnd them which are minima. intersecting the constraint circle.98 −0. we must be on .5 0. the entire enterprise of Lagrange multipliers (which are coming soon. and so the gradient of . we can project onto .5 −1 −0.5 −0. increasing but staying on .

and . it normal of at . we still wanted that there be no direction of increase inside the feasible region. but note that the Lagrangian is a function of +1 variables ( plus ) and so we do have the right number of equations to hope for unique. .+) quadrant. Before we do anything numeric. existing solutions: from the partial derivatives. we can write the Lagrangian. we have and we want We can see from the ﬁrst two equations that . means . If . then . To recap: in the presence of a constraint. . we must have either or . With multiple constraints . at a point where the line is tangent to a level curve of .From the ﬁrst two equations. This occured whenever the gradient at . this method still works perfectly well. since they sum to one. we knew we were at a local extreme because the gradient of was zero – there was no local direction of motion which increased . . If . Let’s say we instead want the constraint that and sum to 1 ( ). while probably not zero. with. In unconstrained optimization (no constraints). and solve for zero.4 More Dimensions If we want to have mutliple constraints. and . These are the minimum and maximum. Actually solving that system of equations can be hard. The last statement generalizes to multiple constraints. because we were conﬁned to . convince yourself from the picture that the maximum is going to occur in the (+. Also convince yourself that the minimum will not be deﬁned. though it get harder to draw the pictures to illustrate it. At those values. 2. plus one from the partial derivative. had no components which were perpendicular to the does not have to be zero at a solution . let’s think of the parallel-normal idea in a slightly different way. which. respectively. To generalize. So what do we have so far? Given a function and a constraint. We will also want the gradient to be non-zero along the ih (I & 0 ¥ he£ ¡ & ¢ # 2m0Y¤¡ $ & £ ¡ # 2 t0 2 0 3" 2 0 3" S TP ¢ # ¡ R £ r ¥ £ P R gP f £ P ¥ £ I P ih f ¨kgP 0 " R £ I h¦£ 0 ¥ 0 § £ ¤¡ 0 § £ ¤¡ 0 § £ ¤¡ d & 2 0 on ¡ $ ¥ e£ d R d£ d which gives S 0 ¥ 2 ge¦£ a0 R £ 2 30 S hP STP R I P ¥ VI P £ r ¥e£¤ F£ ¡ R R £ & P £ '¡ '¢ £¡ 0 " S P vEX 2 0 £ 3'¡ 2 0 ¥ £ UseD r £ I P & S vP R 2 t0 R £ I P R 2 U0 R D £ I P 0 B # ¥ e£ P £ r ¥R £ f gP £¡ ¤¢ # I P S hP X £ 2 0 tB 0 H 0 § £ %f'¡ 0 § £ %f'¡ 0 § £ %f'¡ d 0 § £ f'¡ # 0 0 § £ ¤¡ 2 0 ¥ Use£ ¥ £d R d£ d d d d d (16) (17) (18) (19) (20) (21) (22) (23) (24) (25) (26) (27) which gives the equations ih ¨kf S $ £ l $ £ 0 B a0 R £ l I P 0 " l & ih (jS S 0 R £ & 0 B . Along came the constraint and dashed all hopes of the gradient being completely zero at a constrained extreme . then . differentiate. Then. and . just has to be entirely contained in the (one-dimensional) subspace spanned by . we will insist that a solution satisfy each . that values get arbitrarily low in both directions along the line away from the maximum. Formally. we have the situation in ﬁgure ??(right). However.

or. given the constraints. The sphere grows. where is zero. it is parallel to an (equal) combination of the two planes’ normal vectors. and 0 0 S hP $ l 0 s & $ P £¡ $ '$ t '¢ £¡ S hP £¡ ¤$ $ 0 $ £ X & # # % $ p p 0 £¡ r£ q¦¤¢ $ t . At the points of contact. Those points would be solutions if that plane were the only constraint. 4 In and . When the sphere reaches a radius of . in the “nice” case. and we want to ﬁnd points where all dimensions have zero gradient. Every time we add a constraint. it is touching both planes along their line of intersection. it lies inside the plane spanned by those vectors (the plane . while differentiating with respect to the recovers the assertion that the gradient of have no components which aren’t spanned by the constraints normals. However. of course. so we have less freedom in maximizing. we restrict one dimension. However. and everything should work out. that constraint also removes a dimension along which the gradient must be zero. equivalently. shown as planes in the ﬁgure. When the sphere’s radius reaches one. Again imagine the spheres as expanding from the center. However. our condition should again be that . or subtler situations u S hP s & ¡$ # l S hP X Figure 5: A spherical level curve of the function with two constraint planes. Constraints may be unsatisﬁable (e. and increases. [not shown due to my lacking matlab skills]).g. the sphere touches both planes individually. Note that the gradient is not zero at that point. 0 £¡ ¦¤¢ # I v 0 § £ ¤¡ § £ %f'¡ 2 0 t"£ ¢ # ¡ £ 2 0 £ q¦¤¡ $ & ¢ # ¡ l (28) (29) . Imagine that is the distance from the origin. there are dimensions for to vary along when maximizing. with weights .directions that is free to vary. Thus. nor is it perpendicular to either surface. Let’s say we want the minimum of subject to the constraints that and . So. the gradient of is perpendicular to the touching plane. The unconstrained minimum is. Before there are any constraints. can prevent the Lagrange multipliers from existing [more]. Therefore. A good way to think about the effect of adding constraints is as follows. As an example of multiple constraints. we should be able to add as many or few constraints (up to ) as we wish.4 w @ ws6 G @ "s6 the “not-nice” cases. We can express this by the equation be a linear combination of the normals. until it makes contact with the planes. at the origin. is entirely contained in the subspace spanned by the normals. Which asserts that It turns out that tossing all the constraints into a single Lagrangian accomplishes this: It should be clear that differentiating with respect to and setting equal to zero recovers the th constraint. . all sorts of things can go wrong. the level surfaces of are concentric spheres with the gradient pointing straight out of the spheres. while not necessarily zero. consider ﬁgure ??. cannot make any local movement along vectors which have any component perpendicular to any constraint.

becomes a function of alone. It might not be clear why that’s any different that ﬁxing and ﬁnding a minimizing value for each .2 Reversing the Scope The problem with the above view of the Lagrangian is that it really doesn’t accomplish anything beyond encoding the constraints and handing us back the same problem we started with: ﬁnd the maximum value of . which we aren’t going to prove rigorously. Once we ﬁx the values of the multipliers. You move around looking for a maximum value of . This is a good fact to keep in mind. plus one for each of the ). the intuition behind why it’s true is important. The equations from differentiating with respect to each recovers our gradient conditions. Formally. we would get a solution for each . Therefore. So will be the maximum value of subject to the constraints. . it is often the case that we can ﬁnd a closed-form solution to while we have nothing useful to say about . regardless of what is. to solve. The equations from differentiating with respect the recover the constraints . and the result still holds: This is part of the full Kuhn-Tucker theorem (cite). when you choose . So the numbers give us some conﬁdence that we have the right number of equations to hope for point solutions. 3.) Second. can be ﬁddled to make as small as desired. we can switch the and from the previous section. First. We would very much like for it to become an unconstrained optimization problem. More usefully. you have no control over . This view is easy to understand (but doesn’t really get us anywhere). unlike . [cites] 3. . However. some . Before we examine why this reversal should work. It’s helpful to have an idea of what the Lagrangian actually means. (Remember that it’s negative inﬁnity almost everywhere and jumps to for which satisfy the constraints. But then. regarless of the value of the multipliers. Whenever the constraints are satisﬁed. would not be continuous. problem is to ﬁnd the which gives Now remember that if your x happens to satisfy the constraints. However. described below. call it . let’s see what it accomplishes if it’s true. It’s different in two ways. However. If so. But then we can do an unconstrained minimization of over the space of .3 The Lagrangian The Lagrangian is a function of variables (remember that .3 Duality Let’s say we’re convinced that it would be a good thing if § £ f'¡ ¨ (32) £ £ | y£ £ '¡ £ ¦¤¡ ¡ £ § £ f'¡ $ £¡ 0 § £ '¢m`¤¡ z Ur $ & § £ %f'¡ ¨¤¡ 7 l $ £ $ § £ %f¤¡ ¤¡ ¨ § £ %f¤¡ ¨¤¡ z {r & nz l $ ~ & 2 0 £ ¡ '$ £ £ 0 § £ ¤¡ 7¤¡ ( %¡ £ £ £¡ '¢ £ '¡ %¡ £ $ % eP £¡ 0 § £ '¢3}%f¤¡ g| $ z & £ '¡ $ $ $ dr ¦¤¢xx%f¤¡ y £¡ 0 § £ 0 F 0 § £ %f'¡ ¡ £ £ £ $ l £ z £ (30) (31) . You could imagine using the Lagrangian to do constrained maximization in the following way. the worst way possible for you. Differentiating gives the corresponding equations. we can look at the Lagrangian as an encoding of the problem. We might be able to maximize that function (it’s unconstrained!) relatively easily. ignoring the values of which are not in the feasible region. We originally had a constrained optimization problem. the are zero.1 The Lagrangian as an Encoding First. which gets set in the is chosen to minimize . 3. and will be . each set to zero. This is also a general instance of switching to a dual problem when a primal problem is unpleasant in some way. if does not satisfy the constraints. and so at these point. We would then have our solution. There are two intuitions.

Another way to remember this intuitively is that is probably not zero. then suddenly the gradient of is no longer completely cancelled out by . Let’s form the Lagrangian. be . is at a local minimum. At . Another way of thinking of this is that the gradient of (the top arrow ﬁeld) is balanced at that point by the scaled gradient of the constraint (the second arrow ﬁeld down). Recall the nobrainer problem: maximize subject to . where . At . intuition second. But if has any component which is not in the space spanned by the constraint normals . because the linear combination weights are unique. that isn’t possible. and can therefore be written as a (unique) linear combination of these normals. the maximum value of the Lagrangian is at . and which is a local maximum among the points satisfying the constraints. R £ S TP 0 i£ P VI § £ ih ih ¨¨S (I %¡ £ S 0 £ ¦¤¡ 0 § £ ¤¡ ¡ q0 %§f¤¡ £ S 3l R %£ P VI 0 £¡ 1¦¤¢ S (33) 0 "£ & 2 0 ¡ a} $ ¤¢ £¡ eP S © D2 r ¢ # ¡ ih f ¨kgP £ '¡ . Since each . The straight dark line is the value of at constraint is satisﬁed and so. which can only be larger than . has no effect on and . if we set it to zero (a huge nudge). then we can nudge in a direction inside the allowed region. If we nudge up to . so is a local maximum along that line. may not be zero. ﬁxing and allowing to vary. 3. where .4 A sort-of proof of sort-of the Kuhn-Tucker thereom . So is in the space spanned by the constraint normals . . At that value. . Thus. Let be that combination. The gradient of is not zero. formal proof elsewhere. which is the maximum (and only) value of among the point(s) satisfying the constraint. But is non-zero. Its maximum value (2) is at the origin (which obviously doesn’t satisfy the constraint). The value was . cannot still be zero. We can see the effect by adding these two ﬁelds. except for where . Figure 7 shows what happens when we nudge up and down slightly. which forms the gradient of the Lagrangian (third arrow ﬁeld). and so we can £ i f (h gP S 0 i %¡ £U0"£ S 0 £ ¡S 0 § £ c'¢V¤¡ Hz 0 t£ 0 w X & ¡ $ & # $ $ y Y ¢ # 0 ¡ $ # ¡ ¢ ¡ & ¡$ # ¢ # ¡ $ % & i f ¨h gP % # £ S TP S TP i¨kf g0n h 2 ih ih ¨jS (I X 0y § ux ¤¡ ¡ 0 § £ ¢ ¡ £¡ 0 2§ £ '¢mYf'¡ ¡ £ f ¡ & $ P %§ ¡ # $ # % & $ y ¢P # 0}%§ ¡ # ¡ ¡ ¡ $ # $% $ y ¢ # & % P ¡ 2U ¡ $ # $ $ y ¢ # 0 ¡ ¢ # ¡ I £¡ ¤ 2 t0 P 0 n§ P VI S Figure 6: The Lagrangian of the paraboloid with the constraint 2 30 £ R %£ . Therefore. Since is a local minimum inside that region. increasing . the derivatives of with respect to each are zero. This gradient is zero at with the right . The minimum value along this line is at . The curving dark line is the value of at for all . At each . The maximum value occured at . let’s consider what happens in the environment of a point The Lagrangian is hard to plot when which satisﬁes the constraints. Recall the paraboloid (ﬁgure 5) with the constraint that and sum to one. examples ﬁrst. and. the Lagrangian is just the original surface . there is some direction (either or the reverse direction where we could nudge to increase .10 5 0 −5 −10 −15 −20 −25 −30 −35 4 2 0 0 −2 −4 −4 −2 2 4 Now. the This surface is plotted in ﬁgure ??. as promised. Then clearly . However. (which does satisfy the constraints). and so the maximum of is the unconstrained maximum of . Let’s look another more example. we’ll argue why this is true. Now consider a vector near . at . but it is perpendicular to the constraint line.

Therefore. So it is the rate at which we could increase the Lagrangian if we were to raise the target of that constraint (from zero).5 What do the multipliers mean? A useful aspect of the Lagrange multiplier method is that the values of the multipliers at solution points often has some signiﬁcance.6667 2 lambda = −1. bribery. In economics. Similarly. [Physics example?] 4 A bigger example than you probably wanted This section contains a big example of using the Lagrange multiplier method in practice. . mining. $ & u $ & . if we nudge down to .3333 2 lambda = −1 0 0 0 0 −2 −2 2 0 2 −2 −2 2 0 2 −2 −2 2 0 2 −2 −2 2 0 2 0 0 0 0 −2 −2 2 0 2 −2 −2 2 0 2 −2 −2 2 0 2 −2 −2 2 0 2 0 0 0 0 −2 −2 2 0 2 −2 −2 2 0 2 −2 −2 2 0 2 −2 −2 2 0 2 0 0 0 0 −2 −2 0 2 −2 −2 0 2 −2 −2 0 2 −2 −2 0 2 increase the lagrangian by nudging toward the origin.lambda = 0 2 2 lambda = −1. a multiplier is the value of the partial derivative of with respect to the constraint . This rate is called the shadow price of . when is a proﬁt function and the are constraints on resource amounts. But remember that at solution points . 3. as well as another case where the multipliers have an interesting interpretation. or whatever means). which is interpreted as the amount it would be worth to relax that constraint upwards (by R&D. would be the amount (possibly negative!) by which proﬁt would rise if one were allowed one more unit of resource . u S © D2 S P vEX $ P i f g(h gP ¢a§ ¡ ¡ 0 XI (P P VI $ Figure 7: Lagrangian surfaces for the paraboloid with the constraint 2 t0 r £ R R %£ . then the gradient of is over-cancelled and we can increase the Lagrangian by nudging away from the origin. Mathematically. the rate of increase of the Lagrangian with respect to that constraint is also the rate of increase of the maximum constrained value of with respect to that constraint.

For inequality constraints. then nothing should change at if we drop ). The two possibilities are shown in ﬁgure ??.4 0. is referred to as complementary slackness. except that we require that and that. at any given . a has either or . Constraints of this form are written . out of the feasible side and towards the forbidden side). it allows to be maximum if the gradient of . If then is said to be active at .2 0 4. Recall that we used to write for a (single) equality constraint. is either zero or pointing towards negative values of (which violate the constraint). .1 0.1 Maximum Entropy Models 5 Extensions 5. complete with multiple constraints. we add them to the Lagrangian just as if they were equality constraints. This situation can be compactly written as . In fact. but it must actually point in the opposite sense along that direction (i. (Intuitively. then we want to be even stricter about what values of are acceptable from a solution. The interpretation was that.8 1 1 0. The key observation about inequality constraints work is that.6 0. then there is no reason that we cannot move in that direction.e. then if is non-zero. in this case.2 0.6 0. # $ £ ¦¤¡ & # t¦¤¢ # 0 £¡ # t¦¤¢ # 0 £¡ $ t P £ £¡ '¢ # £ 2 tW £¡ ¦¤¢ # £ '¡ 0 § § £ ff¤¡ # & # '¡ £ 2 '¡ £ £ £ 2 & $ £ ¦¤¡ 2 0 £ U}¤¡ 2 0 £ V'¡ 2 0 £ 3' ¡ 2 £ q¤¡ $ & (34) £ (36) . otherwise it is inactive. If is inactive at ( ).4 0.. if the gradient is pointing towards positive values of . Positive mutlipliers mean that the direction of increasing is in the same direction as increasing – but points in that situation certainly aren’t solutions.6 0. we get the general Lagrangian: 2 0 £ 3'¡ The Kuhn-Tucker theorem (or our intuitive arguments) tell us that if a point the constraints and . as we want to increase and we are allowed to increase .2 0 0 0.1 Inequality Constraints The Lagrange multiplier method also covers the case of inequality constraints. then is a lot like an equality constraint. then: is a maximum of subject to (37) £ '¡ £ £ ¤¡ £¡ '¢ # £ & # £ ¦¤¡ £ P & $ P ¦¤¡ £ (t ¦¤$ % t ¦¤¢3ff¤¡ £¡ $ £¡ 0 § § £ # (t ¦¤¡¢ # £ S s P £¡ '$ £ X Figure 8: The simplex 0 r £ ¦¤¡ r £ £ '¡ & . which are qualitatively very different. .4 0. we write must be entirely in the (35) but.8 0. Bundling it all up. must be zero at .8 0. direction of the normal to . [better explanation] In summary. but not both. The situation that one or the other can be non-zero. by restricting the multiplier to be negative (or zero). if is inactive. However. We can actually enforce this very simply. if is a solution. if x is a maximum. for inequality constraints. then is. it not only has to be parallel . If is active at . if is not zero.

The above conditions are called the ﬁrst-order conditions. The theorem also gives second order conditions on the second derivative (Hessian) matrices which distinguish local maxima from other situations which can trigger “false alarms” with the ﬁrst-order conditions. The third condition is a somewhat cryptic way of insisting that for each . Now is probably a good time to point out that there is more to the Kuhn-Tucker theorem than the above statement.The second condition takes care of the restriction on active inequalities. However. Feedback requested! £ ¦¤¡ $ $ 2 2 0'¡ £ t u $ § ¡ u (38) (39) . The goal of this tutorial was to supply some intuition behind the central ideas so that other. one knows in advance that the solution will be a maximum (such as in the maximum entropy example). If you are interested. Caveat about globals? 6 Conclusion This tutorial only introduces the basic concepts of the Langrange multiplier methods. All (local) maxima will satisfy them. there are many detailed texts on the subject [cites]. either is zero or is zero. in many situations. more comprehensive and formal sources become more accessible.

Stock Cash Financial Perspective

OpenStack Report

SSRN-id1123882

Study_IB

Iot

The Accelerator and Incubator Ecosystem in Europe

IM Mistakes

M1 Basics in Statistics

American Casinos

Due Diligence Checklist

Preapering Financial Statements

MIT Technology Review Business Report the Internet of Things NI (1)

Coal 101 - UBS (2009)

Commodities Primer - RBS (2009)

Broadband Wireless Access

Amazon

yoox

Entropic Communications, Inc. Presents at J.P. Morgan Global Technology, Media & Telecom Conference, May-21-2014 12-00 PM

45472394 Nomura Global Outlooks 2011

UBS

Anadigics, Inc., Q1 2014 Earnings Call, Apr 30, 2014

Nitto Europe Financieel Rapport 2013

yoox

France

cf

- Read and print without ads
- Download to keep your version
- Edit, email or read offline

Are you sure?

This action might not be possible to undo. Are you sure you want to continue?

CANCEL

OK

You've been reading!

NO, THANKS

OK

scribd