Linear Algebra: Review

Anubha Gupta, PhD.

Professor
SBILab, Dept. of ECE,
IIIT-Delhi, India
Contact: anubha@iiitd.ac.in; Lab: http://sbilab.iiitd.edu.in
 Machine Learning in Hindi

Linear Algebra: Review

(Vector Spaces)
 Machine Learning in Hindi

Motivation of studying linear algebra

● Linear Regression: A supervised learning algorithm that uses linear algebra (LA) to model
the relationship between dependent and independent variables. It can be solved using
matrix algebra, which involves finding the inverse of a matrix and multiplying matrices.
● Support Vector Machines (SVMs): A type of supervised learning algorithm that uses LA
to separate data samples into different classes. SVMs find the hyperplane that maximally
separates the data samples by solving a quadratic optimization problem using LA.
● Neural Networks: A type of ML algorithm that are inspired by the structure and function
of the human brain. These use LA to represent the weights between neurons and to
perform the calculations of forward and backward propagation through the network.

● Principal Component Analysis (PCA): A method used for

dimensionality reduction and feature extraction that uses LA to
find the eigenvectors and eigenvalues of the covariance matrix
of the data.
 Machine Learning in Hindi

Learning Objectives
● In this section, we will study the concepts related to the vector spaces. In particular,
we will study
○ Group
○ Field
○ Vector space
○ Metric
○ Norm
○ Inner product
○ Basis vectors
○ Linear independence of basis vectors
○ Span
○ Basis
 Machine Learning in Hindi

Notations
● Scalars will be written as small case letters in italics: x
● Vectors will be written as small case letters in bold: x
● Matrices will be written as upper case letters in bold: X
● The (i,k)th component of a matrix X will be written as Xik
● Single random variable will be written as upper case letter in italics: X
● Vector random variable will be written as upper case, italics, and bold: X
● Complex conjugation of a variable x will be denoted as: 𝑥ҧ
● Transpose of a vector will be represented using superscript T: xT
● Complex conjugate transpose of a vector will be represented using superscript
H: xH
 Machine Learning in Hindi

Group
A group (G, ∗) consists of the following:
(i) A set G
(ii) A rule or binary operation ‘∗’ (set G is closed under this operation) which associates with
each pair of elements x and y in G, an element (x ∗ y) in G such that
a. This binary operation is associative:
i.e., x ∗(y ∗ z)=(x ∗ y) ∗ z ∀ x, y, z ∈ G.
b. There exists an element ‘e’ called identity of group G s.t. e ∗ x = x ∗ e = x ∀ x ∈ G
c. For every x ∈ G, there exists an element x-1 such that
x ∗ x-1 = e = x-1 ∗ x ∀x∈G
d. If x ∗ y=y ∗ x ∀ x, y ∈ G
Commutative group or Abelian group
 Machine Learning in Hindi

Examples
1. Set of n x n invertible matrices with matrix multiplication
2. Set of periodic signals with time period T under binary operation ‘∗’ = ‘+’
 Machine Learning in Hindi

Field
A field (F, +, ·) consists of the following:
(i) A set F
(ii) Two binary operations ‘+’ and ‘·’ such that
a. (F, +) is an Abelian group
b. Define F*=F-{0}. (F*, ·) is an Abelian group
c. The multiplication operation distributes over addition:
● Left distributive:
x·(y+z)=x·y + x·z ∀ x, y, z ∈ F
● Right distributive:
(x+y)· z=x·z + y·z ∀ x, y, z ∈ F

Examples: Check for F = R; F = C; F= Z; F=Q

 Machine Learning in Hindi

Examples
F = R, C and Q form a field, while Z does not form a field.
 Machine Learning in Hindi

Vector Space
A set V together with a map
‘+’: V x V → V
(v1,v2) → (v1+v2) called vector addition
And ‘.’ : F x V → V
(α,v) → (α·v) called scalar multiplication
is called a F-vector space or vector space over the field F if the following are satisfied:
(i) (V, +) is an Abelian group
(ii) α·(v1+v2) = α · v1 + α · v2 ∀ v1, v2 ∈ V and ∀ α ∈ F
(iii) (α1 + α2) · v = α1 · v + α2 · v ∀ v ∈ V and ∀ α1, α2 ∈ F
(iv) (αβ) · v = α(βv) = β(αv) ∀ v ∈ V and ∀ α, β ∈ F
(v) 1·v = v ∀v∈V
(vi) α·0 = 0 ∀α∈F
(vii) 0·v=0 ∀v∈V
(viii) If v≠ 0, then α·v = 0 implies that α = 0.
(ix) If V is a vector space over a field F, then any linear combination
of vectors lying in V (with scalars from F) would again lie in V.
 Machine Learning in Hindi

Vector Space
Examples:
1. Set of periodic signals with the same time period T over the field F= R form a vector space
2. Set of finite energy signals over the field F= R form a vector space
3. Vn(F) = n-dimensional vector space over the field F = R or C
 Machine Learning in Hindi

Examples
Vn(F) = n-dimensional vector space over the field F = R or C
 Machine Learning in Hindi

Metric Space
Metric is a map
d: X x X → R
that satisfies the following:
(i) d(x, y) ≥ 0 and d(x, y)=0 iff x = y ∀ x, y ∈ X
(ii) d(x, y) = d(y, x) ∀ x, y ∈ X
(iii) d(x, y) ≤ d(x, z) +d(z, y) ∀ x, y, z ∈ X

This map is called a metric and a set equipped with this map is called a metric
space and is denoted by (X, d).

Note that metric is a generalization of the notion of distance.

 Machine Learning in Hindi

Examples
Euclidean Distance

d(x, y) ≥ 0 and d(x, y)=0 iff x = y

d(x, y) = d(y, x)

d(x, y) ≤ d(x, z) +d(z, y)

 Machine Learning in Hindi

Norm
Let V be a F-vector space. A map
║∙║: V → R is called a norm if it satisfies the following:
(i) ║v║≥ 0 and ║v║= 0 iff v=0 ∀v∈V
(ii) ║αv║= │α│║v║ ∀ v ∈ V and ∀ α ∈ F
(iii) ║v1+ v2 ║≤ ║v1║+ ║v2║ ∀ v1 , v 2 ∈ V

A vector space equipped with a norm is called a normed vector space.

Example: Let V be a F-vector space equipped with a norm.

Prove that d(v1, v2 ) = ║v1- v2║ is a proper metric.
 Machine Learning in Hindi

Examples
Let V be a F-vector space equipped with a norm. Prove that d(v1, v2 ) = ║v1- v2║ is a proper
metric.
Proof:
d(v1, v2 ) ≥ 0 and d(v1, v2 )=0 iff v1 = v2

d(v1, v2 ) = d(v2, v1)

d(v1, v2 ) ≤ d(v1, v3) +d(v3, v2 )

 Machine Learning in Hindi

Inner Product
Let V be a F-vector space.
A map
< , > : V x V→ F
is called an inner product if it satisfies the following:
(i) < v, v > ≥ 0 and < v, v > = 0 iff v = 0 ∀v∈V
(ii) < v1, v2 > = < 𝐯2 , 𝐯1 > ∀ v1 , v2 ∈ V
‘¯’ denotes the complex conjugate operation.
(i) It is linear in the first coordinate.
< α1v1+ α2v2, w> = α1< v1, w> + α2<v2, w> ∀ v1, v2, w ∈ V and ∀ α1, α2 ∈ F
(i) It is conjugate linear in the second coordinate.
<v, α1w1+ α2w2 > = 1 < v, w1> + 2 <v, w2>
∀ w1, w2, v ∈ V and ∀ α1, α2 ∈ F
 Machine Learning in Hindi

Inner Product
Example 1: Define ║v║= < v, v>1/2. Show that it is a proper norm.
Thus, if a space is an inner product space, we can define norm and then define metric.

Example 2: Consider Vn(F), where vectors are n-tuples of scalars, i.e., x ∈ V and xi ∈ F
i.e.,

Define inner product as:

Show that it satisfies all the properties of an inner product.

 Machine Learning in Hindi

Linear Independence of Vectors

𝑛
A set of vectors {v1, v2, ….., vn} in Vn(F) is called linearly independent (LI) if ෍ 𝑖 𝐯𝑖 = 0
𝑖=1
implies αi = 0 for all i.

Definition: The number of maximal LI vectors in a vector space V is called the dimension
of the vector space and the maximal LI vectors is called a basis for V.

If {e1, e2, ….., en} is a basis for Vn(F) , then for any v ∈ V and for some αi ∈ F
𝑛

𝐯 = ෍ 𝑖 𝐞 𝑖
𝑖=1
 Machine Learning in Hindi

Orthogonal and Orthonormal Basis

Definition: A set of basis vectors {v1, v2, ….., vn} spanning an inner product space V
is called orthogonal basis if
(i) vi ≠ 0 ∀ i
(ii) < vi , vj > = 0 ∀ i≠j

If additionally, the below condition is satisfied:

(iii) < vi , vj > = 1 ∀ i=j

Then, the set of basis vectors is called as the orthonormal basis. In this case, we can
also write the above conditions as < vi , vj > = δij.
 Machine Learning in Hindi

Summary
● In this section, we studied the concepts of
○ Group
○ Field
○ Vector space
○ Metric
○ Norm
○ Inner product
○ Basis vectors
○ Linear independence of basis vectors
○ Span
○ Basis

with examples.
 Machine Learning in Hindi

Acknowledgment
For this set of slides, we would like to thank Prof. Shiv Dutt Joshi, Dept. of Electrical
Engineering, IIT-Delhi (https://iitd.irins.org/profile/10546)