Professional Documents
Culture Documents
Winner PDF
Winner PDF
• Table of Contents
• Begin Article
c 2010 rfuller@abo.fi
November 4, 2010
Table of Contents
1. Kohonen’s network
1. Kohonen’s network
!1 !m
w1n wm1
w11 wmn
x1 x2 xn
The learning
The learning algorithm
algorithm treatstreats of m
the setthe setweight
of mvectors
weightas
Toc JJ II J I Back J Doc Doc I
Section 1: Kohonen’s network 5
kx − wr k = min kx − wi k.
i=1,...,m
kx − wi k2 =< x − wi , x − wi >
=< x, x > −2 < wi , x > + < wi , wi >
= kxk2 − 2 < w, x > +kwi k2
= kxk2 − 2 < wi , x > +1,
w2
w3 w1
of x on the direction of wi .
It is clear that the closer the vector wi to x the bigger the pro-
jection of x on wi .
Note that < wr , x > is the activation value of the winning neu-
ron which has the largest value neti , i = 1, . . . , m.
When using the scalar product metric of similarity, the synap-
tic weight vectors should be modified accordingly so that they
become more similar to the current input vector.
With the similarity criterion being cos(wi , x), the weight vector
lengths should be identical for this training approach. However,
their directions should be modified.
Thus,
kx − wr k
must be reduced, preferebly along the gradient direction in the
weight space
wr1 , . . . , wrn .
= −2(x1 , . . . , xn )T + 2(w1 , . . . , wn )T
= −2(x − w).
It seems reasonable to reward the weights of the winning neu-
ron with an increment of weight in the negative gradient direc-
tion, thus in the direction x − wr .
We thus have
wr := wr + η(x − wr )
where η is a small lerning constant selected heuristically, usu-
ally between 0.1 and 0.7.
The remaining weight vectors are left unaffected.
• Step 1 wr := wr + η(x − wr ), or := 1,
(r is the winner neuron)
• Step 2
wr
wr :=
kwr k
(normalization)
• Step 3 wi := wi , oi := 0, i 6= r (losers are unaf-
fected)
w2
w3 w1
In the end of the training process the final weight vectors point
to the centers of gravity of classes.
Toc JJ II J I Back J Doc Doc I
In the end of the training process the final weight
Section 1: Kohonen’s
vectorsnetwork
point
to the centers of gravity of classes. 14
w2
w3
w1
Figure 4: The final weight vectors point to the center of gravity of the
The final weight vectors point to the center of
classes.
gravity of the classes.
• Linear:
f (< w, x >) = wT x
• Piecewise linear:
1 if < w, x > > 1
f (< w, x >) = < w, x > if | < w, x > | ≤ 1
−1 if < w, x > < −1
• Hard:
f (< w, x >) = sign(wT x)
Toc JJ II J I Back J Doc Doc I
17
• Unipolar sigmoidal:
f (< w, x >) = 1/(1 + exp(−wT x))
• Bipolar sigmoidal (1):
f (< w, x >) = tanh(wT x)
• Bipolar sigmoidal (2):
2
f (< w, x >) = −1
1 + exp(wT x)
2
f (t) = − 1.
1 + exp(−t)
Then the following equality holds
2 exp(−t) 1
f 0 (t) = 2
= (1 − f 2 (t)).
(1 + exp(−t)) 2
• If f is a unipolar sigmoidal activation function of the form
1
f (t) = .
1 + exp(−t)
Then f 0 satisfies the following equality
f 0 (t) = f (t)(1 − f (t)).
2exp(−t)
f 0 (t) = .
[1 + exp(−t)]2
Toc JJ II J I Back J Doc Doc I
Section 3: Derivatives of sigmoidal activation functions 21
2
1 − exp(−t)
1 2exp(−t)
1− =
2 1 + exp(−t) [1 + exp(−t)]2
we get
2exp(−t) 1
2
= (1 − f 2 (t)).
[1 + exp(−t)] 2