Are you sure?
This action might not be possible to undo. Are you sure you want to continue?
ON THE REPRODUCING KERNEL HILBERT SPACE
AND THE DISCRETIZED TIKHONOV REGULARIZATION
ÐÀO MA
.
NH KHANG  VÕ ANH KHOA
Ho Chi Minh City  July 7, 2013
Contents
Contents 1
1 REPRODUCING KERNEL HILBERT SPACES 2
1.1 INTRODUCTION. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 EXAMPLES OF REPRODUCING KERNELS AND HISTORICAL REMARKS. . . . . . 5
2 TIKHONOV REGULARIZATION AND THE REPRESENTER THEOREM 6
2.1 TIKHONOV REGULARIZATION. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Bibliography 7
1
Chapter 1
REPRODUCING KERNEL
HILBERT SPACES
1.1 INTRODUCTION.
Deﬁnition 1.1. Let H be a real Hilbert space of functions deﬁned on Ω ⊂ R
n
with the inner product
(., .)
H
. A function Φ : Ω×Ω →R is called a reproducing kernel for H if the two followings are satisﬁed:
1. For every x ∈ Ω, Φ
x
(y) = Φ(y, x) as a function of y belongs to H.
2. For every x ∈ Ω and every f ∈ H, then f (x) = (f, Φ
x
)
H
.
Remark. The second is called the reproducing property as well.
By applying the second one to the function Φ
y
at f, we get
Φ
y
(x) = (Φ
y
, Φ
x
)
H
, for x, y ∈ Ω.
And then using the ﬁrst one,
Φ(x, y) = (Φ
y
, Φ
x
)
H
= (Φ
x
, Φ
y
)
H
, for x, y ∈ Ω. (1.1.1)
This is called symmetric property of a reproducing kernel Φ.
Deﬁnition 1.2. A Hilbert space H is called a reproducing kernel Hilbert space (RKHS) if there exists a
reproducing kernel Φ of H. And its norm is symbolized by .
H
.
Moreover, for all f ∈ H in the form of f =
N
k=1
α
k
Φ
x
k
with x
k
∈ Ω, α
k
∈ R, its norm is deﬁned by
f
2
H
=
N
k=1
N
j=1
α
k
α
j
Φ(x
k
, x
j
) . (1.1.2)
The following theorem formally establishes the relationship between the RKHS and a reproducing
kernel.
Theorem 1.3 (Moore  Aronszajn). The reproducing kernel of a RKHS is uniquely determined. Con
versely, a symmetric positive deﬁnite reproducing kernel deﬁnes a unique RKHS.
2
CHAPTER 1. REPRODUCING KERNEL HILBERT SPACES 3
Proof. To prove the ﬁrst statement, we assume that there exists Φ
x
and Ψ
x
are simultaneously
determined on a RKHS H.
At this point, using the second of (1.1), we have
(f, Φ
x
)
H
= (f, Ψ
x
)
H
, for f ∈ H.
It means that
(f, Φ
x
−Ψ
x
)
H
= 0, for f ∈ H.
Choosing f = Φ
x
−Ψ
x
, we obtain Φ
x
−Ψ
x
H
= 0. Hence, the reproducing kernel is unique.
For the second statement, given Φ we can construct the RKHS H
0
as the completion of space of
functions spanned by the set {Φ
x
x ∈ Ω} with an inner product deﬁned as follows:
Given two functions f and g in {Φ
x
x ∈ Ω}, we set
f (x) =
N
k=1
α
k
Φ
x
k
(x) , (1.1.3)
g (x) =
N
k=1
β
k
Φ
x
k
(x) , (1.1.4)
we deﬁne their inner product to be
(f, g)
H
0
=
N
k=1
N
j=1
α
k
β
j
_
Φ
x
k
, Φ
x
j
_
H
0
=
N
k=1
N
j=1
α
k
β
j
Φ
_
x
k
, x
j
_
. (1.1.5)
Thus for all x ∈ Ω, we have
(f, Φ
x
)
H
0
=
N
k=1
α
k
(Φ
x
k
, Φ
x
)
H
0
=
N
k=1
α
k
Φ(x
k
, x) = f (x) . (1.1.6)
This means that H
0
has the reproducing property. So the deﬁnition of the inner product above does
not depend on the representations of the functions f and g in H
0
. Since Φ is positive deﬁnite, (f, f)
H
0
≥ 0
for all f ∈ H
0
and we obtain the Cauchy  Schwartz for (f, f)
H
0
. In addition, if (f, f)
H
0
= 0 and then
by (1.1.6), for all x ∈ Ω,
f (x) ≤ f
H
0
Φ
x
H
0
= 0,
which implies that f ≡ 0. Thus,
_
H
0
, (., .)
H
0
_
is a pre  Hilbert space.
(continue)
According to Mercer’s theorem that can be found in [1], we obtain the following theorem.
Theorem 1.4. Let ϕ be in L
2
(Ω) with a compact metric space Ω, we deﬁne
T
Φ
(ϕ) (x) =
ˆ
Ω
Φ(x, y) ϕ(y) dy. (1.1.7)
Then there is an orthonormal basis {e
j
} of L
2
(Ω) consisting of eigenfuntions of T
Φ
such that the
CHAPTER 1. REPRODUCING KERNEL HILBERT SPACES 4
eigenvalues λ
j
are non  negative. The eigenfunctions corresponding to non  zero eigenvalues are con
tinuous on Ω and Φ can be represented as
Φ(x, y) =
∞
j=1
λ
j
e
j
(x) e
j
(y) , (1.1.8)
and the convergence of the sum is uniform on Ω ×Ω, and absolute for each pair (x, y) ∈ Ω ×Ω.
Proof. Firstly, we show that the operator T
Φ
is well  deﬁned, i.e., T
Φ
(ϕ) is a continuous function
for all ϕ ∈ L
2
(Ω). Indeed, for all x, y ∈ Ω, we have
T
Φ
(ϕ) (x) −T
Φ
(ϕ) (t) =
¸
¸
¸
¸
ˆ
Ω
(Φ(x, y) −Φ(t, y)) ϕ(y) dy
¸
¸
¸
¸
= (Φ
x
−Φ
t
, ϕ)
L
2

≤ Φ
x
−Φ
y
L
2
ϕ
L
2
≤
_ˆ
Ω
(Φ(x, y) −Φ(t, y))
2
dy
_
1/2
ϕ
L
2
≤
√
Ωmax
y∈Ω
Φ(x, y) −Φ(t, y) ϕ
L
2
.
Since Φ is a continuous function on a compact domain and ϕ is in L
2
(Ω), we obtain T
Φ
(ϕ) is a
continuous function on Ω.
Secondly, ...
Theorem 1.5. Let Ω be a compact metric space and H
1
deﬁned by
H
1
=
_
_
_
f ∈ L
2
(Ω) :
∞
j=1
λ
−1
j
¸
¸
¸(f, e
j
)
L
2
(Ω)
¸
¸
¸
2
< ∞
_
_
_
, (1.1.9)
and the inner product can be written as
(f, g)
H1
=
∞
j=1
λ
−1
j
(f, e
j
)
L
2
(Ω)
(g, e
j
)
L
2
(Ω)
, f, g ∈ H
1
. (1.1.10)
Then H
1
= H (they are the same spaces of functions with the same inner product).
Proof. Firstly, we show that (1.1.10) deﬁnes an inner product. Indeed, we easily realize that it is
linear, symmetric and (f, f)
H
1
≥ 0 since λ
j
are non  negative. Moreover, (f, f)
H
1
= 0 if and only if
f ≡ 0.
For the second step, by (1.1.8), we have Φ
x
=
∞
j=1
λ
j
e
j
(x) e
j
. Thus we see that
∞
i=1
λ
−1
i
¸
¸
¸(Φ
x
, e
i
)
L
2
(Ω)
¸
¸
¸
2
=
∞
i=1
λ
−1
i
¸
¸
¸
¸
¸
¸
∞
j=1
λ
j
e
j
(x) (e
j
, e
i
)
L
2
(Ω)
¸
¸
¸
¸
¸
¸
2
=
∞
i=1
λ
i
e
i
(x)
2
< ∞,
which implies that Φ
x
∈ H
1
for all x ∈ Ω.
Furthermore, since f ∈ L
2
(Ω), we probably have the form f =
∞
i=1
a
i
e
i
with a
i
∈ R. Then, by the
deﬁnition of its inner product,
CHAPTER 1. REPRODUCING KERNEL HILBERT SPACES 5
(f, Φ
x
)
H
1
=
∞
j=1
λ
−1
j
_
∞
i=1
a
i
e
i
, e
j
_
L
2
(Ω)
_
∞
k=1
λ
k
e
k
(x) e
k
, e
j
_
L
2
(Ω)
=
∞
j=1
λ
−1
j
a
j
e
j
λ
j
e
j
(x)
= f (x) .
Thus, H
1
is a Hilbert space of functions with a reproducing kernel Φ, so it must be equal to the
original H by the uniqueness of RKHS.
Conclusion. The outcome of the above theorem is that although space H
1
is deﬁned using the integral
operator T
Φ
and its associated eigenfunctions {e
j
} which depend on some prerequisite conditions, it still
coincides exactly with the original RKHS H of Φ by the uniqueness of RKHS.
1.2 EXAMPLES OF REPRODUCING KERNELS AND HIS
TORICAL REMARKS.
RKHS were explicitly introduced in learning theory by F. Girosi (1997). However, this idea was used
for the ﬁrst time at the beginning of the 20th century by S. Zaremba in his work on boundary value
problems for harmonic and biharmonic functions. But he did not develop this theory after that.
For a long time these results were not investigated and seemed to be forgotten. Then the idea once
again appeared in some papers and works of S. Bergman, S. Bochner, E. H. Moore, N. Aronszajn,
S. Saitoh and so on in many subjects, for instance, approximation theory, computer vision, partial
diﬀerential equations, probability theory.
For now, they are used to deal with many important problems, and can be considered as a nearly
new tool in Mathematics.
In statistical problems, people frequently use the three following kernels.
• Linear kernel
Φ(x, y) = x · y,
• Gaussian kernel
Φ(x, y) = e
−
x−y
2
σ
2
, σ > 0,
• Polynomial kernel
Φ(x, y) = (x · y + 1)
n
, n ∈ N.
Chapter 2
TIKHONOV REGULARIZATION
AND THE REPRESENTER
THEOREM
2.1 TIKHONOV REGULARIZATION.
The goal of regularization is to restore the well  posedness (speciﬁcally, making the result depend
smoothly on the data) of the empirical risk minimization technique by eﬀectively restricting the hypoth
esis space H.
Let E be an arbitrary set, and let H
Φ
be the RKHS admitting a reproducing kernel Φ(., .) on E.
For any Hilbert space H we consider a bounded linear opterator K : H
Φ
→H. We shall study the best
approximate problem
inf
f∈H
Φ
Kf −b
H
, (2.1.1)
for a vector b in H.
6
Bibliography
[1] Holger Wendland, Scattered Data Approximation, Cambridge University Press, 2005.
[2] Y. C. Hon, Tomoya Takeuchi, Discretized Tikhonov regularization by reproducing kernel Hilbert space
for backward heat conduction problem, Adv Comput Math (2011) 34 : 167  183.
7
This action might not be possible to undo. Are you sure you want to continue?