MA423 Matrix Computations

Lecture 10: Vector and matrix norms

Rafikul Alam
Department of Mathematics
IIT Guwahati

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Outline

• Vector norms
• Matrix norms
• Unitarily invariant norms

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Vector norms

Let V be a vector space over C. Then a function k · k : V → R is called a

norm on V if it satisfies the three fundamental properties:
(a) Positive definiteness: kv k ≥ 0 and kv k = 0 ⇐⇒ v = 0.
(b) Positively homogeneous: kαv k = |α| kv k for α ∈ C and v ∈ V.
(c) Triangle inequality: ku + v k ≤ kuk + kv k for u, v ∈ V.

Example: Consider Cn and the vector norms given by

1-norm: kxk1 := |x1 | + · · · + |xn |.
p
2-norm: kxk2 := |x1 |2 + · · · |xn |2 .
∞-norm: kxk∞ := max1≤j≤n |xj |.

More generally, for 1 ≤ p < ∞, the Höder p-norm is given by

kxkp := (|x1 |p + · · · + |xn |p )1/p .

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Matrix norms

Let A ∈ Cm×n . Then A : Cn −→ Cm , x 7−→ Ax, is a linear map. Suppose

Cn and Cm are equipped with norms. Then

kAxk
kAk := max = max kAxk
x6=0 kxk kxk=1

defines a norm on Cm×n and is called an induced matrix norm or a

subordinate matrix norm.

For the identity matrix kIxk = kxk and hence kI k = 1. Note that

kAxk ≤ kAk kxk

for all x ∈ Cn .

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Matrix norms
The matrix norm of A induced by the Hölder p-norm is denoted by kAkp .
Then kAk1 , kAk2 and kAk∞ are called 1-norm, 2-norm and ∞-norm of
A, respectively. Also kAk2 is called the spectral norm of A.

Theorem: Let A be an m × n matrix. Then

kAk1 = max kAej k1 = max kA(:, j)k1

1≤j≤n 1≤j≤n
p
kAk2 = λmax (A∗ A)
kAk∞ = max kei> Ak1 = max kA(i, :)k1 ,
1≤i≤m 1≤i≤m

where λmax (A∗ A) denotes that largest eigenvalue of A∗ A.

Proof: We have
Ax = x1 Ae1 + · · · + xn Aen ⇒ kAxk1 ≤ max1≤j≤n kAej k1 kxk1 . This yields
kAk1 ≤ max1≤j≤n kAej k1 . But kAej k1 ≤ kAk1 for all j = 1 : n. Hence we
have kAk1 = max1≤j≤n kAej k1 . 

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Spectral norm

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Frobenius norm

The spectral norm of a matrix is difficult to compute. By contrast,

Frobenius norm is easy to compute and provides a nice geometric
framework for solving problems. Let A be an m-by-n matrix.

Then the Frobenius norm is given by

 1/2
Xm X
n p p
kAkF :=  |A(i, j)|2  = Tr(A∗ A) = Tr(AA∗ ),
i=1 j=1

where Tr(A∗ A) is the trace of A∗ A.

Pm
1/2 1/2
P
> 2 n
Note that kAkF = i=1 kei Ak2 kAej k22
= . Also note
j=1
√
that for an n-by-n identity matrix I , we have kI kF = n which shows
that the Frobenius norm is not an induced matrix norm.

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Example

Let D := diag(λ1 , . . . , λn ) be a diagonal matrix. Then we q

have
Pn 2
kAk2 = kAk1 = kAk∞ = max(|λ1 |, . . . , |λn |) and kDkF = j=1 |λj | .
   
1000 999 −998 999
Consider A := . Then A−1 = .
999 998 999 −1000

Thus kAk∞ = kAk1 = kA−1 k∞ = kA−1 k1 = 1999.

On the other hand,

kAk2 = 1998.000500500375 and kAkF = 1998.000500500438.

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Submultiplicative matrix norm
A matrix norm is said to be submultiplicative if

kABk ≤ kAk kBk

holds for all A and B. An induced matrix norm is submultiplicative.

Indeed, kABxk ≤ kAk kBxk ≤ kAk kBk kxk =⇒ kABk ≤ kAk kBk.

The Frobebnius norm is also submultiplicative. Indeed, suppose A is

m-by-n and B is n-by-p matrices. Then (AB)ij = ei> ABej shows that
|(AB)ij | ≤ kei> Ak2 kBej k2 for i = 1 : m and j = 1 : n. Consequently,

X p
m X p
m X
X
kABk2F = |(AB)ij |2 ≤ kei> Ak22 kBej k22
i=1 j=1 i=1 j=1
m
X p
X
= kei> Ak22 kBej k22 = kAk2F kBk2F .
i=1 j=1

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Unitarily invariant matrix norm
Let Q ∈ Rn×n . Then Q is said to be orthogonal if Q > Q = QQ > = I . If
Q is orthogonal
p then p p
kQxk2 = hQx, Qxi = hx, Q > Qxi = hx, xi = kxk2 .

Let Q ∈ Cn×n . Then Qpis said to be unitary

p if Q ∗ Q =pQQ ∗ = I . If Q is
unitary then kQxk2 = hQx, Qxi = hx, Q ∗ Qxi = hx, xi = kxk2 .

It follows that kQk2 = 1 when Q is unitary and that error in x is NOT

magnified in the vector Qx.

A matrix norm is said to be unitariliy invariant if kQAPk = kAk holds for

any unitary matrices Q and P. The spectral norm and the Frobenius
norm are unitarily invariant norms. Indeed,

kQxk2 = kxk2 ⇒ kQAPk2 = kAk2 .

p p
kQAPkF = Tr((QAP)∗ QAP) = Tr(P ∗ A∗ AP)
p
= Tr(A∗ A) = kAkF .

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC