You are on page 1of 15

# EE263 Autumn 2007-08 Stephen Boyd

Lecture 8
Least-norm solutions of undetermined
equations

## • general norm minimization with equality constraints

8–1
Underdetermined linear equations

we consider
y = Ax
where A ∈ Rm×n is fat (m < n), i.e.,

## • x is underspecified, i.e., many choices of x lead to the same y

we’ll assume that A is full rank (m), so for each y ∈ Rm, there is a solution

## set of all solutions has form

{ x | Ax = y } = { xp + z | z ∈ N (A) }

## Least-norm solutions of undetermined equations 8–2

• z characterizes available choices in solution

## Least-norm solutions of undetermined equations 8–3

Least-norm solution

## i.e., xln is solution of optimization problem

minimize kxk
subject to Ax = y

## Least-norm solutions of undetermined equations 8–4

suppose Ax = y, so A(x − xln) = 0 and

## (x − xln)T xln = (x − xln)T AT (AAT )−1y

= (A(x − xln))T (AAT )−1 y
= 0

## Least-norm solutions of undetermined equations 8–5

{ x | Ax = y }

N (A) = { x | Ax = 0 }
xln

{ x | Ax = y }

## Least-norm solutions of undetermined equations 8–6

• A† = AT (AAT )−1 is called the pseudo-inverse of full rank, fat A

## cf. analogous formulas for full rank, skinny matrix A:

• A† = (AT A)−1 AT

## Least-norm solutions of undetermined equations 8–7

Least-norm solution via QR factorization

## find QR factorization of AT , i.e., AT = QR, with

• Q ∈ Rn×m, QT Q = Im

then

## • xln = AT (AAT )−1y = QR−T y

• kxlnk = kR−T yk

## Least-norm solutions of undetermined equations 8–8

Derivation via Lagrange multipliers

## • least-norm solution solves optimization problem

minimize xT x
subject to Ax = y

## • optimality conditions are

∇xL = 2x + AT λ = 0, ∇λ L = Ax − y = 0

## Least-norm solutions of undetermined equations 8–9

Example: transferring mass unit distance

## • y = Ax where A ∈ R2×10 (A is fat)

• find least norm force that transfers mass unit distance with zero final
velocity, i.e., y = (1, 0)

0.06

0.04

xln
0.02

−0.02

−0.04

−0.06
0 2 4 6 8 10 12

1
t
0.8
position
0.6

0.4

0.2

0
0 2 4 6 8 10 12

0.2
t
velocity

0.15

0.1

0.05

0
0 2 4 6 8 10 12

## Least-norm solutions of undetermined equations 8–11

Relation to regularized least-squares

## • suppose A ∈ Rm×n is fat, full rank

• define J1 = kAx − yk2 , J2 = kxk2
• least-norm solution minimizes J2 with J1 = 0
• minimizer of weighted-sum objective J1 + µJ2 = kAx − yk2 + µkxk2 is

T
−1
xµ = A A + µI AT y

## • fact: xµ → xln as µ → 0, i.e., regularized solution converges to

least-norm solution as µ → 0
• in matrix terms: as µ → 0,

T
−1 T T T −1

A A + µI A →A AA

## Least-norm solutions of undetermined equations 8–12

General norm minimization with equality constraints

consider problem
minimize kAx − bk
subject to Cx = d
with variable x

## • includes least-squares and least-norm problems as special cases

• equivalent to
minimize (1/2)kAx − bk2
subject to Cx = d

• Lagrangian is

## L(x, λ) = (1/2)kAx − bk2 + λT (Cx − d)

= (1/2)xT AT Ax − bT Ax + (1/2)bT b + λT Cx − λT d

## Least-norm solutions of undetermined equations 8–13

• optimality conditions are

∇xL = AT Ax − AT b + C T λ = 0, ∇λL = Cx − d = 0

T T T
    
A A C x A b
=
C 0 λ d

   T T
−1  T

x A A C A b
=
λ C 0 d

## Least-norm solutions of undetermined equations 8–14

if AT A is invertible, we can derive a more explicit (and complicated)
formula for x

## • from first block equation we get

x = (AT A)−1(AT b − C T λ)

## • substitute into Cx = d to get

C(AT A)−1(AT b − C T λ) = d

so
T −1 T −1 T −1 T
 
λ = C(A A) C C(A A) A b−d

## • recover x from equation above (not pretty)

 
T −1 T T T −1 T −1 T −1 T

x = (A A) A b−C C(A A) C C(A A) A b−d