You are on page 1of 16

# Objectives_template

## Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

## The Lecture Contains:

Singular Value Decomposition (SVD)
SVD of real symmetric matrix
Transformation using SVD
Example
Example: compact form

## Dimensionality reduction using SVD

Example of dimensionality reduction (k = 1)

## Compact way of dimensionality reduction (k = 1)

How many dimensions to retain?

Objectives_template

## Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

## Singular Value Decomposition (SVD)

Factorization of a matrix

If A is of size

Columns of

, then

is

is

and

is

matrix

are eigenvectors of

## Left singular vectors

(orthonormal)

Columns of

are eigenvectors of

(orthonormal)

## are the singular values

is diagonal
Singular values are positive squareroots of eigenvalues of
(assuming

or

singular values)

Objectives_template

## Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

## SVD of real symmetric matrix

A is real symmetric of size

since

is of size

## This is called spectral decomposition of

contains
Eigenvectors of

singular values
= eigenvectors of

Eigenvalues of

= squareroot of eigenvalues of

Eigenvalues of

= singular values of

Objectives_template

## Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

Transformed data

## is called SVD transform matrix

Essentially,

is just a rotation of

Dimensionality of

is

Columns of

## give the basis vectors in rotated space

Objectives_template

## Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

Transformed data

## is called SVD transform matrix

Essentially,

is just a rotation of

Dimensionality of

is

Columns of

## shows how each dimension can be represented as a linear combination of other

dimensions
Columns are input basis vectors
shows how each object can be represented as a linear combination of other objects
Columns are output basis vectors

Objectives_template

## Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

Transformed data

## is called SVD transform matrix

Essentially,

is just a rotation of

Dimensionality of

is

Columns of

## shows how each dimension can be represented as a linear combination of other

dimensions
Columns are input basis vectors
shows how each object can be represented as a linear combination of other objects
Columns are output basis vectors
Lengths of vectors are preserved

Objectives_template

## Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

Example

Objectives_template

## Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

If

is of size

, then

is

is

and

is

matrix

## non-zero singular values in

Objectives_template

## Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

Use only
Retain first
First

dimensions
columns for

columns of

Best rank

and

and first

values for

## approximation in terms of sum squared error

Objectives_template

## Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

## Example of dimensionality reduction (

= 1)

Objectives_template

## Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

## Compact way of dimensionality reduction (

= 1)

Objectives_template

## Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

## How many dimensions to retain?

Objectives_template

## Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

## How many dimensions to retain?

Objectives_template

## Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

## How many dimensions to retain?

Concept of energy of a dataset
Total energy is sum of squares of singular values (aka spread or variance)

Retain

Generally,

is between

## of the energy is retained

to

Objectives_template

## Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

## How many dimensions to retain?

Concept of energy of a dataset
Total energy is sum of squares of singular values (aka spread or variance)

Retain

Generally,

is between

## of the energy is retained

to

= 1 retains

of the energy

Objectives_template

## Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

## How many dimensions to retain?

Concept of energy of a dataset
Total energy is sum of squares of singular values (aka spread or variance)

Retain

Generally,

is between

to

= 1 retains

Running time:

for

of size

of the energy

and rank