Lesson 13: Newton in several variables

restart;

Jacobians and Vector functions

There are very big and capable LinearAlgebra and VectorCalculus packages that can do lots of
things with Vectors and Matrices, but we won't need to go much beyond the very basic level of
Vector and Matrix operations from Lesson 12.

I'll need one command from the VectorCalculus package: Jacobian.

Consider a column Vector V whose components are expressions depending on several variables.
V := <f(x,y,z), g(x,y,z), h(x,y,z)>;

(1.1)

The Jacobian of F is the Matrix of partial derivatives of the components of F with respect to each of
the variables. Since this is the only member of the VectorCalculus package I'll want to use today,
and I want to avoid some side-effects of VectorCalculus, I'll only take this one procedure from
VectorCalculus. You can do that with an extra input to with.
with(VectorCalculus, Jacobian);
(1.2)
J := Jacobian(V,[x,y,z]);

(1.3)

Notice that the rows of J correspond to the components of V, and the columns correspond to the
variables. You could think of each row of J as the gradient vector of a component of V.
It will be convenient for us to consider functions from Vectors to Vectors. You can define these like
this:
F := X -> <X[1] + X[2]*X[3], X[2] - X[1]^2, cos(X[3])>;
(1.4)

F(<a,b,c>);

(1.5)
 (1.5)

Unfortunately there isn't an equivalent of the D operator to differentiate such a function: Jacobian
has to be applied to a Vector of expressions, not to a function such as F. What you can do is this:
Jacobian(F(X),[X[1],X[2],X[3]]);

(1.6)

And then you can make this into a function using unapply.
J:= unapply(%,X);
(1.7)

J(<a,b,c>);

(1.8)

J(<1,2,3>);

(1.9)

Why am I using unapply here rather than ->?

badJ:= X -> Jacobian(F(X),[X[1],X[2],X[3]]);

This would work if the components of X were variables:

badJ(<x,y,z>);

but not with constants:

badJ(<1,2,3>);
Error, (in badJ) invalid input: too many and/or wrong type of
arguments passed to VectorCalculus:-Jacobian; first unused
argument is [1, 2, 3]
 The Jacobian command needs to take an expression involving variables and differentiate it with
respect to variables. Here badJ is calling Jacobian with a constant vector F(<1, 2, 3>) and a list of
constants [1, 2, 3]. Maple complains, and rightly so. On the other hand, my definition of J
computed the Jacobian once, using variables, and then used that to define a function that can be
given either variables or constants.

Newton's method in several variables

So far we've seen Newton's method used for solving one equation in one variable. Now I want to
look at the extension of this to solving a system of equations in several variables.
The basic idea of Newton's method is the same:
Start with an initial guess .
Consider the linear approximation to near the initial guess.
Find the point where the linear approximation is 0.
Use that as the next guess. Repeat this process until it appears to converge.
The linear approximation to near is where is the Jacobian
matrix for at
Thus in two variables:
F:= X -> <f(X[1],X[2]), g(X[1],X[2])>;
(2.1)

J := unapply(Jacobian(F(X),[X[1],X[2]]),X);
(2.2)

F(<a[1],a[2]>) + J(<a[1],a[2]>) . (<x[1],x[2]>-<a[1],a[2]>);

(2.3)

Now to find where this is 0. Assuming the Matrix J(A) is invertible, the solution of
is
Let's try an example.
sys := [x^2 + x*y*z + y^2*z^3 = 3,
x*y^2 - y*z^2 - 2*x^2 = 0,
x^2*y + y^2*z + z^4 = 4];
(2.4)
I want to write this as , where F is a vector-valued function. I'll subtract the right sides
from the left:
map( lhs - rhs, sys);
(2.5)
Change x, y, z to the components of the vector X.
eval(%,{x=X[1],y=X[2],z=X[3]});
(2.6)
Make this into a column Vector.
Vector(%);

(2.7)

Use unapply to make this into a function of X.

F:= unapply(%,X);
(2.8)

Now to get the Jacobian:

J := unapply(Jacobian(F(X),[X[1],X[2],X[3]]),X);
(2.9)

The Newton iteration will be

Newt:= X -> evalf(X - J(X)^(-1) . F(X));
(2.10)

For example, if we take this starting point

W[0]:= <1, 2, 3>;

(2.11)

Newt(W[0]);

(2.12)
 (2.12)

Did that bring us closer to a solution?

F(%);

(2.13)

F(W[0]);

(2.14)

Maybe... Let's try a few more iterations.

for count from 1 to 10 do
W[count]:= Newt(W[count-1])
end do;
 (2.15)

It seems to have pretty much settled down (except for roundoff error).
F(W[10]);

(2.16)

So it seems there is a solution here. Is it the only one? Let's try a different initial condition.
Y[0]:= <0,0,1>;
for count from 1 to 10 do
Y[count]:= Newt(Y[count-1])
end do;

Error, (in rtable/Power) singular matrix

Oops...
LinearAlgebra[Determinant](J(<x,y,z>));
(2.17)
Note each term has at least two of the variables x, y, z. So to get the Jacobian to be nonsingular, you
need at least two of x, y, z to be nonzero. Try again...
Y[0]:= <1,0,1>;
for count from 1 to 10 do
Y[count]:= Newt(Y[count-1])
end do;
 (2.18)

Let's try to get a picture.

with(plots):
The implicitplot3d command in the plots package is similar to implicitplot, but in three dimensions
instead of two, and without gridrefine or crossingrefine options. It does have the grid option (with
a list of three integers), but high numbers greatly increase the time needed to plot.
implicitplot3d(sys,x=-10..10,y=-10..10,z=-10..10,colour=[red,
blue,green],axes=box);

implicitplot3d(sys,x=0..2,y=0..2,z=0..1,colour=[red,blue,
green],axes=box,style=patchnogrid,lightmodel=light4);
Here's my small contribution to Maple. The intersectplot command, also in the plots package, plots
in 3 dimensions the curve where two surfaces intersect. Here's the intersection of the first two
surfaces.
intersectplot(sys[1],sys[2],x=-10..10,y=-10..10,z=-10..10,
colour=red, axes=box);
Here are the intersections of all three pairs of surfaces. At the intersection points, all three curves
should meet.
display([%,intersectplot(sys[2],sys[3],x=-10..10,y=-10..10,z=
-10..10,colour=blue),
intersectplot(sys[1],sys[3],x=-10..10,y=-10..10,z=-10..10,
colour=green)],axes=box); IP1:= %:
It would be nice to see the solutions we found earlier in here. The pointplot3d command in the
plots package can be used to plot a list of points in 3 dimensions. While we're at it, I'll also zoom in
a bit using the view option.
display([IP1, pointplot3d([W[10],Y[10]],symbol=solidbox,
symbolsize=10, colour=black)],view=[-3..3,-3..3,-3..3]);
How many solutions are there in all? Newton won't tell us. Maybe solve will.
solve(sys);

(2.19)
Somewhat surprisingly, we need Digits to be pretty high to get reasonably accurate numerical values
of the solutions. Even 20 wasn't enough.
evalf([allvalues(%)],25);
(2.20)
(2.20)
sols:=remove(has,%,I);
(2.21)

pts:= [seq(eval(<x,y,z>,sols[i]), i = 1..nops(sols))];

(2.22)

W[10], Y[10];

(2.23)

display([IP1, pointplot3d(pts,symbol=solidbox, symbolsize=10,

colour=black)],view=[-3..3,-3..3,-3..3]);
Parametric curves
We've seen several ways of plotting curves in Maple. For curves in the plane of the form
you can use

plot(f(x), x=a..b);

For curves given by an implicit equation such as you can use implicitplot in the plots
package.

implicitplot(f(x,y)=0, x=a..b, y=c..d);

A third kind of curve is a parametric curve, where x and y are given as functions of a parameter t, e.g.
. For this you can use

plot([f(t), g(t), t=a..b]);

For example:
 plot([cos(t),sin(t),t=0..2*Pi]);
1

0 1

In three dimensions an "implicit" curve would have to be given by two equations in the variables x,y,
z, and intersectplot can be used for this:

intersectplot(equation1, equation2, x=a1 .. b1, y=a2 .. b2, z = a3

.. b3);

You can also have a parametric curve in 3 dimensions. For this you can use spacecurve in the plots
package.
spacecurve([x(t),y(t),z(t)], t=a..b);
For example:
spacecurve([sin(t),cos(t),sin(2*t)],t=0..2*Pi, colour=green,
axes=box);
How close is "close enough"?
If you start Newton's method at a point close enough to a solution, it should converge rapidly to that
solution. But how close? We investigated that for Newton's method in one variable. What about
several variables? I'll look at a two-variable example, because the pictures will be easier to draw.
F:= X -> <X[1]^2 * X[2] + X[1]*X[2]^2 - 2,
X[1] + sin(X[2]) - 2>;
(4.1)

J:= unapply(Jacobian(F(X),[X[1],X[2]]),X);
(4.2)

W[0]:= <1, 2>;

for count from 1 to 10 do
W[count]:= Newt(W[count-1])
 end do;

(4.3)

Here are the curves corresponding to the two equations.

with(plots):
implicitplot([F(<x,y>)[1], F(<x,y>)[2]],
x = 0 .. 4, y = -4 .. 4, colour = [red,blue]);
 4

y 0
1 2 3 4
x

Clearly there are two solutions in this rectangle, of which we've found one with Newt. I'll store that
and look for the other one.
Sol1:= W[10];
W[0]:= <2,-3>;
for count from 1 to 10 do
W[count]:= Newt(W[count-1])
end do;
 (4.4)

Sol2:=W[10];

(4.5)

Sol1,Sol2;

(4.6)

Could there be more outside this rectangle? Take another look at the equations:
F(<x,y>) = <0,0>;

Since is always between -1 and 1, it's clear from the second equation that x must be between
1 and 3. Now the first equation is a quadratic in y, so any vertical line can only intersect its curve in
at most two points. But for x between 1 and 3, we see both of those two points in our graph (one on
each of the red curves). So there can't be any other solutions.
Now to investigate our question: how close is "close enough"?
One way to proceed is to plot circles of various radii centred at the solution together with their
images under Newt.
Suppose we find that for every radius with 0 < , the image of the circle is always inside a
slightly smaller circle. Then, starting from any point whose distance from the solution is less than
 , you will always get convergence to the solution.
I'll plot two parametric curves. One (in red) is the circle of radius R = 0.1 centred at the solution we
found, the point Sol1, and the other (in blue) is the image of that under Newt.
R:= 0.1:
plot([[Sol1[1] + R*cos(t), Sol1[2] + R*sin(t),
t = 0 .. 2*Pi],
[ Newt(Sol1+<R*cos(t),R*sin(t)>)[1],
Newt(Sol1+<R*cos(t),R*sin(t)>)[2],
t = 0 .. 2*Pi]], colour=[red,blue]);

The blue curve is the image of the red circle under Newt, and it's well within it.

Maple objects introduced in this lesson

LinearAlgebra package
VectorCalculus package
Jacobian (in VectorCalculus package)
Determinant (in LinearAlgebra package)
implicitplot3d (in plots package)
intersectplot (in plots package)
pointplot3d (in plots package)
parametric version of plot
spacecurve (in plots package)