Linear Algebra

Tutorial I
EECS 442
Fall 2020, University of Michigan

Slide Adapted from David Fouhey 1

 Vectors

2 𝒙𝒙1 = 2
𝒙𝒙 =
3 𝒙𝒙2 = 3

x = [2,3] Just an array!

Get in the habit of thinking of
them as columns.

Slide Adapted from David Fouhey 2

 Scaling Vectors

2x = [4,6] • Can scale vector by a scalar

• Scalar = single number
• Dimensions changed
x = [2,3] independently
• Changes magnitude / length,
does not change direction.

Slide Adapted from David Fouhey 3

 Adding Vectors
• Can add vectors
• Dimensions changed independently
• Order irrelevant
• Can change direction and magnitude
x = [2,3] x+y = [5,4]

y = [3,1]

Slide Adapted from David Fouhey 4

 Scaling and Adding

2x+y = [7,7]

Can do both at the same

x = [2,3] time

y = [3,1]

Slide Adapted from David Fouhey 5

 Measuring Length
Magnitude / length / (L2) norm of vector
𝑛𝑛 1/2
2
𝒙𝒙 = 𝒙𝒙 2 = � 𝑥𝑥𝑖𝑖
𝑖𝑖
x = [2,3] There are other norms; assume L2
unless told otherwise
𝒙𝒙 2 = 13
y = [3,1] 𝒚𝒚 = 10
2
Why?
Slide Adapted from David Fouhey 6
 Normalizing a Vector

x = [2,3] Diving by norm gives

something on the unit
sphere (all vectors with
length 1)
𝒙𝒙′ = 𝒙𝒙/ 𝒙𝒙 𝟐𝟐
y = [3,1]
𝒚𝒚′ = 𝒚𝒚/ 𝒚𝒚 𝟐𝟐

Slide Adapted from David Fouhey 7

 Dot Products
𝑛𝑛

𝒙𝒙 ⋅ 𝒚𝒚 = � 𝑥𝑥𝑖𝑖 𝑦𝑦𝑖𝑖 = 𝒙𝒙𝑻𝑻 𝒚𝒚

𝑖𝑖=1
𝒙𝒙 ⋅ 𝒚𝒚 = cos 𝜃𝜃 𝒙𝒙 𝒚𝒚
What happens with
𝒙𝒙′ normalized / unit
𝜃𝜃 vectors?
𝒚𝒚′

Slide Adapted from David Fouhey 8

 Dot Products
𝑛𝑛

𝒙𝒙 = [2,3] 𝒙𝒙 ⋅ 𝒚𝒚 = � 𝑥𝑥𝑖𝑖 𝑦𝑦𝑖𝑖

𝑖𝑖
What’s 𝒙𝒙 ⋅ 𝒆𝒆𝟏𝟏 , 𝒙𝒙 ⋅ 𝒆𝒆𝟐𝟐 ?
Ans: 𝒙𝒙 ⋅ 𝒆𝒆𝟏𝟏 = 2 ; 𝒙𝒙 ⋅ 𝒆𝒆𝟐𝟐 = 3
• Dot product is projection
𝒆𝒆𝟐𝟐 • Amount of x that’s also
pointing in direction of y

𝒆𝒆𝟏𝟏
Slide Adapted from David Fouhey 9
 Dot Products
𝑛𝑛

𝒙𝒙 = [2,3] 𝒙𝒙 ⋅ 𝒚𝒚 = � 𝑥𝑥𝑖𝑖 𝑦𝑦𝑖𝑖

𝑖𝑖
What’s 𝒙𝒙 ⋅ 𝒙𝒙 ?
2
Ans: 𝒙𝒙 ⋅ 𝒙𝒙 = ∑𝑥𝑥𝑖𝑖 𝑥𝑥𝑖𝑖 = 𝒙𝒙 2

Slide Adapted from David Fouhey 10

 Special Angles
1 0
⋅ =0∗1+1∗0=0
0 1
𝒙𝒙 Perpendicular /
orthogonal vectors
have dot product 0
𝒙𝒙′ irrespective of their
𝜃𝜃 magnitude

𝒚𝒚′ 𝒚𝒚
Slide Adapted from David Fouhey 11
 Special Angles
𝑥𝑥1 𝑦𝑦1
𝑥𝑥2 ⋅ 𝑦𝑦2 = 𝑥𝑥1 𝑦𝑦1 + 𝑥𝑥2 𝑦𝑦2 = 0
𝒙𝒙
Perpendicular /
orthogonal vectors
have dot product 0
𝒙𝒙′ irrespective of their
𝜃𝜃 magnitude

𝒚𝒚′ 𝒚𝒚
Slide Adapted from David Fouhey 12
Orthogonal Vectors

𝒙𝒙 = [2,3]
• Geometrically,
what’s the set of
vectors that are
orthogonal to x?
• A line [3,-2]

Slide Adapted from David Fouhey 13

 Orthogonal Vectors
𝒙𝒙 • What’s the set of vectors that are
orthogonal to x = [5,0,0]?
• A plane/2D space of vectors/any
vector [0, 𝑎𝑎, 𝑏𝑏]

• What’s the set of vectors that are

𝒙𝒙 orthogonal to x and y = [0,5,0]?
• A line/1D space of vectors/any
vector [0,0, 𝑏𝑏]
𝒚𝒚 • Ambiguity in sign and magnitude

Slide Adapted from David Fouhey 14

 Cross Product
• Cross product 𝒙𝒙 × 𝒚𝒚 is: (1)
𝒙𝒙 orthogonal to x, y (2) has sign
𝒚𝒚 given by right hand rule and (3)
has magnitude given by area of
parallelogram of x and y
𝒙𝒙 × 𝒚𝒚 • Important: if x and y are the same
direction or either is 0, then 𝒙𝒙 ×
𝒚𝒚 = 𝟎𝟎 .
• Only in 3D!

Slide Adapted from David Fouhey 15

Image credit: Wikipedia.org
 Operations You Should Know

• Scale (vector, scalar → vector)

• Add (vector, vector → vector)
• Magnitude (vector → scalar)
• Dot product (vector, vector → scalar)
• Dot products are projection / angles
• Cross product (vector, vector → vector)
• Vectors facing same direction have cross product 0
• You can never mix vectors of different sizes

Slide Adapted from David Fouhey 16

 Matrices
Horizontally concatenate n, m-dim column vectors
and you get a mxn matrix A (here 2x3)

𝑣𝑣11 𝑣𝑣21 𝑣𝑣31

𝑨𝑨 = 𝒗𝒗1 , ⋯ , 𝒗𝒗𝑛𝑛 = 𝑣𝑣 𝑣𝑣22 𝑣𝑣32
12

Slide Adapted from David Fouhey 17

 Matrices
𝑎𝑎 𝑇𝑇
Transpose: flip
𝑏𝑏 = 𝑎𝑎 𝑏𝑏 𝑐𝑐 (3x1)T = 1x3
rows / columns
𝑐𝑐
Vertically concatenate m, n-dim row vectors
and you get a mxn matrix A (here 2x3)
𝑇𝑇
𝒖𝒖1 𝑢𝑢11 𝑢𝑢12 𝑢𝑢13
𝐴𝐴 = ⋮ = 𝑢𝑢 𝑢𝑢22 𝑢𝑢23
21
𝒖𝒖𝑇𝑇𝑛𝑛
Slide Adapted from David Fouhey 18
 Matrix-Vector Product
𝒚𝒚2𝑥𝑥𝑥 = 𝑨𝑨2𝑥𝑥𝑥 𝒙𝒙3𝑥𝑥𝑥
𝑥𝑥1
𝑦𝑦1
= 𝒗𝒗𝟏𝟏 𝒗𝒗𝟐𝟐 𝒗𝒗𝟑𝟑 𝑥𝑥2
𝑦𝑦2
𝑥𝑥3

𝒚𝒚 = 𝑥𝑥1 𝒗𝒗𝟏𝟏 + 𝑥𝑥2 𝒗𝒗𝟐𝟐 + 𝑥𝑥3 𝒗𝒗𝟑𝟑

Linear combination of columns of A

Slide Adapted from David Fouhey 19

 Matrix-Vector Product
𝒚𝒚2𝑥𝑥𝑥 = 𝑨𝑨2𝑥𝑥𝑥 𝒙𝒙3𝑥𝑥𝑥
3

𝑦𝑦1 𝑻𝑻
𝒖𝒖𝟏𝟏
𝑦𝑦2 = 𝒙𝒙 3
𝑻𝑻
𝒖𝒖𝟐𝟐
𝑻𝑻 𝑻𝑻
𝑦𝑦1 = 𝒖𝒖𝟏𝟏 𝒙𝒙 𝑦𝑦2 = 𝒖𝒖𝟐𝟐 𝒙𝒙
Dot product between rows of A and x

Slide Adapted from David Fouhey 20

 Matrix Multiplication
Generally: Amn and Bnp yield product (AB)mp

− 𝑻𝑻
𝒂𝒂𝟏𝟏 − | |
𝑨𝑨𝑨𝑨 = ⋮ 𝒃𝒃𝟏𝟏 ⋯ 𝒃𝒃𝒑𝒑
− 𝒂𝒂𝑻𝑻𝒎𝒎 − | |
Yes – in A, I’m referring to the rows, and in B,
I’m referring to the columns

Slide Adapted from David Fouhey 21

 Matrix Multiplication
Generally: Amn and Bnp yield product (AB)mp

| |
𝒃𝒃𝟏𝟏 ⋯ 𝒃𝒃𝒑𝒑
| |
− 𝒂𝒂𝑻𝑻𝟏𝟏 − 𝒂𝒂𝑻𝑻𝟏𝟏 𝒃𝒃𝟏𝟏 ⋯ 𝒂𝒂𝑻𝑻𝟏𝟏 𝒃𝒃𝒑𝒑
𝑨𝑨𝑨𝑨 = ⋮ ⋮ ⋱ ⋮
− 𝒂𝒂𝑻𝑻𝒎𝒎 − 𝒂𝒂𝑻𝑻𝒎𝒎 𝒃𝒃𝟏𝟏 ⋯ 𝒂𝒂𝑻𝑻𝒎𝒎 𝒃𝒃𝒑𝒑
𝑨𝑨𝑩𝑩𝑖𝑖𝑖𝑖 = 𝒂𝒂𝑻𝑻𝒊𝒊 𝒃𝒃𝒋𝒋
Slide Adapted from David Fouhey 22
 Matrix Multiplication

• Dimensions must match

• Dimensions must match
• Dimensions must match
• (Yes, it’s associative): ABx = (A)(Bx) = (AB)x
• (No it’s not commutative): ABx ≠ (BA)x ≠ (BxA)

Slide Adapted from David Fouhey 23

 Cross-correlation
Consider 1D case for simplicity
• Correlation c 𝑚𝑚 = ℎ ∗ g = ∑𝑘𝑘 ℎ 𝑚𝑚 + 𝑘𝑘 𝑔𝑔[𝑘𝑘]
• Convolution 𝑓𝑓 𝑚𝑚 = ℎ ∘ g = ∑𝑘𝑘 ℎ 𝑚𝑚 − 𝑘𝑘 𝑔𝑔[𝑘𝑘]
Let ℎ= [3,1,2,5,4],𝑔𝑔 = [1,2,3], then c = [11, 20, 24]:
3 1
c 0 = ∑𝑘𝑘 ℎ 0 + 𝑘𝑘 𝑔𝑔[𝑘𝑘] =ℎ 0 𝑔𝑔 0 + ℎ 1 𝑔𝑔 1 + ℎ 2 𝑔𝑔 2 = 1 � 2
2 3
Each output element is from a dot product!
3 1 2 5 4 3 1
c 0 = 1 � 2 = 3 + 2 + 6 = 11
1 2 3
2 3
1 1
1 2 3 c 1 = 2 � 2 =1 + 4 + 15 = 20
5 3
1 2 3 2 1
c 2 = 5 � 2 =2 + 10 + 12 = 24
4 3
Slide Adapted from David Fouhey 24
 Convolution
Consider 1D case for simplicity
• Correlation c 𝑚𝑚 = ℎ ∗ g = ∑𝑘𝑘 ℎ 𝑚𝑚 + 𝑘𝑘 𝑔𝑔[𝑘𝑘]
• Convolution 𝑓𝑓 𝑚𝑚 = ℎ ∘ g = ∑𝑘𝑘 ℎ 𝑚𝑚 − 𝑘𝑘 𝑔𝑔[𝑘𝑘]
Let ℎ= [3,1,2,5,4],𝑔𝑔 = [1,2,3], then f = [3, 7, 13, 12, 20, 23, 12]:
0 3
f 0 = ∑𝑘𝑘 ℎ 0 − 𝑘𝑘 𝑔𝑔[𝑘𝑘] =ℎ 0 𝑔𝑔 0 + ℎ −1 𝑔𝑔 1 + ℎ −2 𝑔𝑔 2 = 0 � 2
3 1
Each output element is from a dot product!
0 0 3 1 2 5 4 0 0
3 2 1 3

3 2 1 7

3 2 1 13

3 2 1 12
.
Slide Adapted from David Fouhey 25
.
 Operations They Don’t Teach
You Probably Saw Matrix Addition
𝑎𝑎 𝑏𝑏 𝑒𝑒 𝑓𝑓 𝑎𝑎 + 𝑒𝑒 𝑏𝑏 + 𝑓𝑓
+ =
𝑐𝑐 𝑑𝑑 𝑔𝑔 ℎ 𝑐𝑐 + 𝑔𝑔 𝑑𝑑 + ℎ
What is this? FYI: e is a scalar
𝑎𝑎 𝑏𝑏 𝑎𝑎 + 𝑒𝑒 𝑏𝑏 + 𝑒𝑒
+ 𝑒𝑒 =
𝑐𝑐 𝑑𝑑 𝑐𝑐 + 𝑒𝑒 𝑑𝑑 + 𝑒𝑒

Slide Adapted from David Fouhey 26

 Broadcasting
If you want to be pedantic and proper, you expand
e by multiplying a matrix of 1s (denoted 1)
𝑎𝑎 𝑏𝑏 𝑎𝑎 𝑏𝑏
+ 𝑒𝑒 = + 𝟏𝟏2𝑥𝑥𝑥 𝑒𝑒
𝑐𝑐 𝑑𝑑 𝑐𝑐 𝑑𝑑
𝑎𝑎 𝑏𝑏 𝑒𝑒 𝑒𝑒
= +
𝑐𝑐 𝑑𝑑 𝑒𝑒 𝑒𝑒
Many smart matrix libraries do this automatically.
This is the source of many bugs.

Slide Adapted from David Fouhey 27

 Broadcasting Example
Given: a nx2 matrix P and a 2D column vector v,
Want: nx2 difference matrix D

𝑥𝑥1 𝑦𝑦1 𝑥𝑥1 − 𝑎𝑎 𝑦𝑦1 − 𝑏𝑏

𝑎𝑎
𝑷𝑷 = ⋮ ⋮ 𝒗𝒗 = 𝑫𝑫 = ⋮ ⋮
𝑥𝑥𝑛𝑛 𝑦𝑦𝑛𝑛 𝑏𝑏
𝑥𝑥𝑛𝑛 − 𝑎𝑎 𝑦𝑦𝑛𝑛 − 𝑏𝑏

𝑥𝑥1 𝑦𝑦1 𝑎𝑎 𝑏𝑏 Blue stuff is

𝑷𝑷 − 𝒗𝒗𝑇𝑇 = ⋮ ⋮ − ⋮ assumed /
𝑥𝑥𝑛𝑛 𝑦𝑦𝑛𝑛 𝑎𝑎 𝑏𝑏 broadcast
Slide Adapted from David Fouhey 28
 Broadcasting Rule
When operating on two arrays, NumPy compares their
shapes element-wise. It starts with the trailing
dimensions and works its way forward. Two dimensions
are compatible when
1. they are equal, or
2. one of them is 1

Slide Adapted from David Fouhey 29

 Two Uses for Matrices

1. Storing things in a rectangular array (images,

maps)
• Typical operations: element-wise operations,
convolution (which we’ll cover next)
• Atypical operations: almost anything you learned in
a math linear algebra class
2. A linear operator that maps vectors to
another space (Ax)
• Typical/Atypical: reverse of above

Slide Adapted from David Fouhey 30