Ch 2 Vector spaces metric and convergence

21 Vector spaces
Definition A vector space over R the real domain is a set U
together with two functions

veorddinit.VN V ie xty where

is
scalar multiplication or
RN V ie XX
Orsimply xx
where x ER KD
that sagging the flowing
of addition
Associativity xtlytz xtyHZVxy.z.EU
Commuting of
additioni Xty 4⽐ Ex y EV
Zero vector ⺕ an element denoted by 0 in V at

X to 0 X x EXE V

Negative vector UXEV I an element denotedby X.in IV st

xtcx D x 0

VXEV.li X

UXEVandx.PER x px gx
VXEVandxpER.at Pkxxtpx
⽐ y EV and x ER xcxty xxt 对囪
Remark i We can define vector space over G the complex domain similarly
We will assume vector space over R for default Vector space
over G is used very rarely

Example Ii R is a vector space with t the standardaddition of

red numbers and the standard multiplication ofred numbers
Example 2 R is a vector space with t and defined by i
 addition

𡂈㶝㤈 㒡 𨶹 𠠬
scdarmipkativxc.IR and
㓹EM 忄剡 簦
Many input data can be
modeled by vectors in 他
Digitalsound signals of
length n

Time series of length n

n different attributesfeatures of a single thing or object

Example 3 All red mxn matrices is a ve car space with standard

matrix addition and standard scalar multiplication
This rector space is the same as Rmn
An mxn matrix can be used to represent a blackwhite
digital of mxn pixels
image

Example 4 All 3 arrays of Size mxnxl is a vector space

if t and is defined
gtilfxtxijkiiandygj.fi in
xty Xijktyijklii.ie
V xixjhij.in and
XER.mn
x
Xiii i
This rector space is the same as Rmnf
An nxmx33 array can be used to model a color digital

image where Xijk means lip th pixel in channel K and

channel 1，2 了 means Red Green Blue channels of the img

An nxmxl 3 array can be used to mod aig

where
Xijk means the cijtthpixda.tk th frame
Example 5 Consider the set of all strings
Define the addition by e.g
I t am I am
and some scalar multiplication
Then it doesn't form a rector space

Therefore we cannot use vector space to model text data in

this naive way
How to vectorie the texts is a fundamental research topic
in text data analysis and natural language processing
Example
6 The function space Cab f l f is continuous on a的

is a vector space if we define t and by

V it
fgECGH.lftgltkfltjtglD.tt
EGbI.ilffECEabIXER.f
的 JHKXfG.ca
is referred to as a function space since any vector
in the vector space is a function
C 区 b could be the hypothesis space of a learner with
one input and one output ie

X⼀ 两次
with Xi Ea bj and Yi ER
leareafECaiis.t.fi Yi for all i
Example Ti The infinite sequence
⽇ at Cst
laikcoi
l
倒1
With atbli aitbiforallabd.fr all
if x ali xai for all a El for all
X ER
It forms a vector space
This vector space can be used to model e.g
time series with a very long time and or a veryfine time
resolution
 12 Metric in vector spaces
In order to do calculus on vector spaces we need to
define distance closeness between vectors

Let U be vector space Let X y EV Then

a

distance X y distance x y.gg distance x yo

票
distanashouldbeshiltinrariant.TN
length of Ky

Therefore to define a distance we only need to define

a length for each vector in V
Let XE V Let MI be its length called norm which
should satisfy
be nonnegative i.e
a length should
化1120 UXEV
Moreover only the zero vector has a zero length
is
1𠯢 0 x 0
The length of multiple of a vector should be
a

the multiple of the length of the rector.ie

以 ER 11 XXII ⼆ 仪1 ⽐ 11

xxx 7

Triangular inequality the lengthof the direct path is

the smallest x

My 1141x11 ⼗ 州 ynxH
hrnm.HN
o y
 Definition Let U be a rector space over B A norm on V
is a function 11.11 V R such that i
㖄 20 ⽐EV and 侧 0 X 0

HxxlklxlllXH.UXEVandxER
HXtYlKllxlltllyll.lfX.yEV

Example Ii R is a recon space over R

Let ME M HER Then it is a norm on
R
Can you find other norms on R
Example Ii R is a red space over R
There are many norms on Rn
为 2 norm Euclidean norm 化⼀ 贴
11㚲
Èxik
Theinduced distance y
化州12 点 化 州2K is the Euclidean distance
l

normix.MN
点到 ⼆
1
The induced distance in ish
化州 ⼆ 点⻛州 y
is Han distance
known as Mahan You can walk only
horizontally and vertically
o norm
化州
1⽐11 是总 叫
⼆

The induced distance is

N_N ⼆
照⼼ 必 以 1 y
 为 p norm
p D
11Xllp 点 叫啊
Comparison of unit balls

Oi
8 S

mmbaufxln.to
Note that R 11州 112711112 他 州 are all
different normedspaces So given vector space
for a we

can obtain various normed space by choosing different norms

Calculate the norms
of ㄨ 引
⼩b 344怗 ⼆ 5 化11 131 1141 7
侧 ⼆ Max 3 4 4

Example 3 RM is a vector space

Since R can be viewed as RM we can define

vector p norms for matrices
分
HAhear 贰点 19测 ⼆

p 1 sum of absolute values

of entries
of A
p 2 sum of Squares and then take
square root
also known as Frobenius norm
 1
slrobemusnormjllAHF
Foi
fla.jp
A
max entry of in magnitude

Calculate the Frobenius norm of A ⼆ EDER

Mlk ⾔⾔们吖 12124344 丱⼆ 回
AER is also a linear transformation from R lol Rm
Define moms on R and Rm respectively

We can define
nim of a matrix
by
upHAXHP
HAHpp s.CC heck it is a norm

XEnnllxllpxtozsupxM

xllppt_A.UA

so
Find the vector 2 norm Frobenius nom and the matrix
2 norm
of A 69
Frobenius norm 么
HAIf 12 1 0404 ⻔ 冷
matrix 2norm
㖄
嚻𦦵 篼器
⼆
2 1
Example
4 Cab is a vectorspace over R
The 0 vector in Ca bj is thefunction that takes value oona.by
 measure how large a function
I f is
f
we can use the following norms
t
卌 惢⽤刚
andiron
⼆

Then distance twofunctions

of fg ECG b T
g
is Hfgk 器 HM 9叫
Some other norms of Cab can be

All ⼆
fblfcxlldxllflb.fi
If 化性 妒
分
Hfllp fdlf丱测
Example 5i.l.is a vector space over R

U a Elo define 11 印 fail

It is a norm on lo called hi norm

for any infinite sequence a 们 ⽐不

Similarly
分
we define Half
⼆
医⼈们 where
pzl
consider the Nay to Cho
space
lp a

that
We can show lp is a vector space
and Holly is a norm on
lp call lpnorm
Example
a 偐
Then Hall sǐpkil 11

到兰 哥
2
1炶 或 们
Hall 点卡 to
 Therefore
a El a Eh but a Cfl

Remarks i l For the same vector space we can define infinitely many
different norms

technique in machine learning to find

the
z A common

solution is to optimize some norm of the unknown

Different norms lead to very dif rent solutions
 1.2 Case Study Clustering Kmeans k medians

Clustering
Suppose we are given N rectors X Xz ⼀
不 ER
The good of clustering is to group or partition the vectors into
k groups or clusters with the vectors in eachgroup close to each other

oi

We R because it is simple get able to model a variety

use
of
datasets e.g singds images videos attributes of things
the any normal
Actually methods can be extended to spaces
no
Applications
Topic discovery Suppose the N vectors are word histograms

with N documents respectively the jth component in Xi

it
is the counts of the j th word in document i
A clustering algorithm partition the documents into K groups
which typically can be integrated as groups
of documents with
the same topics genre or author
N
Patient during If life feature Vectors associated

with N patients admitted to a hospital a clustering algorithm

clusters the patient into K groups of similar patients
Recommendation system A group of N people respond
to ratings of n movies A clustering algorithm can be used to
cluster the seopole into K groups each with similar taste
 Then we can recommend new movies liked by someone to people in

the samegroup as himher

Many other applications

Mathematical formulation

Representation

Let Ci EG 2 ⼀ K be the group that Xi belongs to i⺦ 2 ⼀

N
Then group j denotedby Gj is Gj ikij3 FI 2 ⼀ ⽔
We assign each group a representative near denoted by Z Zz Z

The representative vectors are not necessarily one

of given vectors
Evaluation
vectors should
First of all within one specificgroup Gj all
be chose to the representative vector Ii MǘpreciseG let
11 不 位
⼆
Jj 到 点
Then Jj Should be small
Second y consider all groups since each Jj is small

J J ⼗ It t Jk
should be small
Altogether we solve the following

器 J 器品 意可 ⼀
点 必 列川
Z Zk
⼀ Zi ZK 暴 意
悲
Optimization

We may use an alternating minimization to solve the minimization

Step Ii Fix the representatives Z ⼀ ⼆ find the best partitions

G GK.ie Son

G照品 杰德 怀 酬 __ ___
1
Step2 Fix the groups G ⼀
Gk find the best representatives
Z Zk i.e Solve

it 新意 Mi 2州 ____
2
The two steps are repeated until convergence

Let's find the solutions of the subproblems CD and 2 respectively

For l
finding the partition G ⼀
Gk is equivalent to
finding a a ⼀
a So 1 becomes
c 只只品 怀 到1411X22岭 ⼀
11 不 到12
nnr tree
nn.ru
dependson depends on depends
a only on Gvonly
Gong
I
min llxi Za.IE i 1.2 N
⼀⼀

Siaa GE 1，2
⼀
k了 to get Ci we only need to compare

NXi Z.IE 11 Xi ZIN HXi Zk.IE

and choose the minimum from it i

e.ci
argmnhxi
jail Zjlli.it 2 ⼀
K

In

qtherwrdszrp
Xi is assigned to thegroup whose representative vector is

For 2
thecbsestoxichrrre
It is rewritten as

只
只 趴意 Milli ⼗ 点 Nit ⼗
点 伈 灭们
tree nny.tt rinse
dependson Z depends on depends on
only ZzOnly ⼆ only

Obviously it is equivalent to minimize each term independently

 ie sohe k independent problems

Mf 意 不可们 抖 2 K

𤄙 ⼼ 难⼆

Èljej
点
⼆ 点 意 Xie
ZjeDEeahte.mn
is this summation are independent again

Thus
等剾以 渺
2 ⼼
嶃
器意
eni.int
One Variable minimization

Eking derivative writ Zjeand setting it to 0 we obtain that

the solution Zje satisfies

2 点 Zi Xie 0
is the numberof
Zje 意如 MG.lk Gjl tents in G
太1，2 n

In vector form

䰘 前 意懰 ⼀ 百⼆ 歃意州
In other 以
of all vectors in Gj
Zj is the mean no 7
Altogether we get the following clustering algorithm

Input Xi K 从 ER
G Gi CN and Zj.FI K
⼀
Output
Initialization Initialize K vectors
Z 瓦 Zk by choosing
 from X ⽕ ⼀
不 randomly

Step Ii Given Zi Zz Zk
⼀
compute

ZjlE
Ci ⼆
agjiii.gl Xi i l 2

N.anddefneGj
Snilci j3 j 1.2 k

step 2i Given G 后 ⼀ Gk compute

Zj 前 意则
Go back to step l

This algorithm is know as K means algorithm because it computes

K means of vectors at step 2

K medians Algorithm

In K means the Euclidean norm is used We can replace it by

mom We solve

in 意愿 不可川
The numerical solver is

Step Ii Fix Zi Zk SoWe

只品
G只
点 意 必 可以
Similar to the discussion in Kmeans the solution is

Ci
aggii.gl Xi Zjlh E1 z

N.andGj
SnilCi
B.Step2
FixG
点您z 炱
Gz ⼀⼀

⼀
Gk
伙 列⼀
solve
 Similar to the discussion in K means it is decomposed

into K sub problems

⼏
步 意11 ⼈ 到1 Fl 2 ⼀
K

It is well known Galileo that the solution is

median Xi
Zj i E Gj
Where median Xi takes component_wise median
i E Gj

This algorithm is called Indian algorithm

 23 Limit and convergence in vector spaces

Convergence in normed vector spaces

To define calculus we need first define convergent sequence

Let U be a normed vector space

Let 廵了KEN be a sequence in V lie Xiv Uk 1.2

Let X EV We say 线圈 converges to X denoted by X_x if

简化 为 11 0

Example Ii Consider R with ll.lk

Let 如 ⼆
關 EN and x
𠇁 0

Then
nxnxh 11⽔㖃 志幽⽕
卡 12122 tn 怍
名
X 112 台品 卡 1424 州
品化
台
constant ink
0
Therefore x 0 ask.se
Example 2 i Consider Co 1 with Mlb

Let f t Sin 2⺎ ⼼ 仁

be the co function
Let 0
Then Pin
2⺎ ⼼
Hfdb 器 仁 六
So 台品 ⾌
点叫 ⽨ 011 0
⼆

f
Therefore 0
 f fi f
Example 3 Consider infinite sequences

叫 酅 篍 噝 a.ee

In b with ll.kz norm

伈 从 112 11a 112

匧 刘兰 K 刘兰卡
台品 kialb 0

Thus ⼼ a in h
In lo with 11 11 norm

119㖄 ⼆
六
Hit all
以
品 优 all
个 0

Thus a a in ho
In l with Holl norm

Ha _all ⼆ Ha 11 ⼆ 点卡 1

㗊㤈 _all 1 to
Thus it a in l
hisexampleshowsth.at
1 7在 ⼼⼼ 如 公以⽣ ⼉所以 0⼏ ⼏⼼ 以
 a is an infinite sequence
Example 4i Consider V a
nan

with 11.11 norm

Let
⼼
鮣 㿶
Then
㗊 niall 品1
冰 珆逾
⼆ ⼆ 0
𤒇
honor m
i.e the limit of ⼼ 我should be a in

However a EV because 㖄 to

but a
4 V because Hall ⼆ 点 10
a normal
0 This example shows that The limit may not be in
vector space
call the normed vector space incomplete
If this happen we

We can always complete a normed vector space by including

all limit of tonerget genes

Completeness of normed vector spaces

Given a sequence X 呢nu in a vector space V with norm M
how to determine that the sequence should be convergent

it 唉
纞 熊 㵘 智品
i.fii
r ii
sk
Lemma
If X X EV then 如 is a cauchy sequence

Proof 奾 U 台 0 ⼆ ks.t.tk 2 K 必以XKÉ

x implies i
Therefore 11㥃北川 EMM xlltllx KE.lt Kl 2K ⽹
The reverse of the lemma is not true Consider
example 4 above where
o

Ha a i a if Kl 2点 1 assume lzy
鯋
But the limit of alki is not in V

Avectorspaavuithnormltllisompkuifnttl
allcauchysequemeinvisonvergenthrre
IAcompletenormedvectorspaaiscalledaBanach
We.am always complete a normed vector space by including
dlimitofauchygnashherg
Example of complete normed vector spaces Banach spaces
norm
R with any
R with any norm
Tensor space RMMM with any
norm.ca
叮 with Nb norm

lp with pzl and finite lo

Example of incomplete normed vector spaces

Example 4 is the last sectioni

V alaisiiiistguithnomM.is incomplete

The completion of this space is lo

ca b with p norm.pt to p21 are incomplete

The completion of this space is done in Red Analysis

Applications

Nang algorithms in data analysis are Iterative algorithms which

generates S sequences of vectors

Tinier EN CV 1.2 S
⼀
s

where V is endowed with a norm 从11

Example ink means algorithm we generate a sequence of

representative vectors fig Zig ka
Initialize ⽨ Zi randomly

for iter 1，2，3 ⼀

given zit no
Zi generate Gif G.ir
given Gif 甅 Gig generate zit

zitertyend.LV
11 11 must

Otherwise if the
be complete
limit of Xi is not in V then the
the limit of Xi may not be a good solution
since V is the hyperthesis spaair the set of good solutions
Example In a supervised learning given Xi 则 ⼼ ⼀
m 發息
We want to find a function
fs.t.flxil yi.ii.im
we aim at finding f E Cia D i.e a continuous function

Assume an algorithm generates a sequence of functions

ffiyiterENCCGbJ.Ifllfliteri
flh sofrsomefin.SI might not be
in Cia 的 lie might not be continuous since abJ.lHh is
incomplete

How to check the sequences Y㘜 is convergent

 g
We can not use the definition of convergence since the limit is
unknown

We use Cauchy sequence to check the convergence

we check whether or not 11如 ⼀如11 is smallenough

刘
In practice this condition is approximated by 化 _xlitery is
small enough

Example In K means algorithm

for iter 1 2 __

__

__

if Aziz 想HEE for all 5 1.2 ⼀ K the stop

end for
24 Finite dimensional rector spaces

In most of the cases we will be dealing with

finitedinensiond vec.tn spaces e.g RI RM RM

which have very nice properties i

dimensional vector space V all norms

For a finite net
are equivalent in the sense that
For any two norms MA and Hb there exists
Guys llalbEGllalh.lkEli
Such that
G.cz so

Consequently t limit of the same sequence under any

norm is the same

如 x in 11𢘛 如 x in Mlb
to prove i 红
狐 0 化以 姤
proof It is equivalent
HXM xlh o OEMM xlbEGIIXMxlb 0
rs 伏以 姤 0

MM xlb 0 soEMM xlh.EE 必 此仍 0

化以 𡚸 0 ⽹

Notethat the constants G.cz depend on V

Example Consider Rn and 1111 11 112 1111
11.11 and ll.lk are equivalent because

MIKE Hall Erm

llalh.ll.lk
and Holl are equivalent

Mlb E Hah E rn Mlb

11011 and 11.11 are equivalent
Halls Hall En Mlb
 speed depends on norms
The convergence

Example
𠱃上
职篚河
摔
X ⼆

x so under any norm

懳开
However 化 以 0112 i
1吣 011 ⼆ 是
To achieve E precision
normikic kz
fl.no
2
⼆
rmi T I
T K E

K is about 泛 times of Kz
dimensional normal vector space is complete
Any finite
lie they are Banach spaces