Competitive Networks: Master Student: Jean Carlo Grandas F. Professor: Carlos Borrás P., PHD., MSC

Competitive Networks
Master Student: Jean Carlo Grandas F.

Professor: Carlos Borrás P., PhD., MSc.
Escuela de Ingeniería Mecánica
Research Group: Dinámica Multifísica, Control y Robótica. DICBoT
November 4, 2020
Master Student: Jean Carlo Grandas F.Professor: Chapter

Carlos Borrás
15 - Presentation
P., PhD., MSc. (Escuela de November
Ingeniería Mecánica
4, 2020 Research
1 / 23 G
Hamming Network
The neurons on the output layer of the Hamming network compete with
each other to determine a winner. The winner indicates which pattern is
most representative of the input pattern.
Hamming network consist of two layers. The first one performs a

correlation between the input and the prototype vectors. The second
performs the competition to determine which of the prototype vectors is
the closest to the input.

Carlos Borrás
15 - Presentation
4, 2020 Research
2 / 23 G
Hamming Network
Layer 1
In order to have multiple inputs to be classified, we need to have multiple
instar. Suppose that we want to recognize the following pattern.

p1 , p2 , ..., pQ
Then the weight matrix W1 , and the bias vector, b, will be:
 T   T  
1w p1 R
 2 w T  p T  R 
  2
W1 =  .  =  .. ,b1 =  . 
  
.
 .   .   .. 
T
pQ T
Sw R
1
Each row of W represents a prototype vector which we want to recognize,
and each element of b1 is set equal to the number of elements in each
input vector (R).

Carlos Borrás
15 - Presentation
4, 2020 Research
3 / 23 G
Hamming Network
Thus the output is,

 T 
p1 p + R
pT p + R 
 2
a1 = W1 p + b1 =  ..


 . 
pT
Qp + R

Carlos Borrás
15 - Presentation
4, 2020 Research
4 / 23 G
Hamming Network
Layer 2
Here, we want to decide which prototype vector is the closest to the input.
Instead of hardlim transfer function, we will use a competitive layer to
chose the closest prototype.
The neurons in this layers are initialized with the outputs of the
feedforward layer, which indicates the correlation between the prototype
pattern and input vector.
After competition, only one vector will have nonzero output.
a2 (0) = a1

Carlos Borrás
15 - Presentation
4, 2020 Research
5 / 23 G
Hamming Network
Then, the second-layer output is updated according to following recurrence

relation:
a2 (t + 1) = poslin(W2 a2 (t))
The second layer weights W2 are set so that diagonal elements are 1, and
off-diagonal elements have a small negative value.

2 1, i = j
wij =
− i 6= j
This produces lateral inhibition, in which the output of each neuron has an
inhibitory effect on all of the other neurons.

Carlos Borrás
15 - Presentation
4, 2020 Research
6 / 23 G
Hamming Network
To illustrate this effect on the outputs,
 
X
ai2 (t + 1) = poslin ai2 (t) − aj2 (t)
j6=i
At each iteration, each neuron’s output will decrease in proportion to the

sum of the other neuron’s output. Only one neuron will have non zero
output.

Carlos Borrás
15 - Presentation
4, 2020 Research
7 / 23 G
Competitive Layer
The second-layer neurons in the Hamming network are said to be in
competition because each neuron excites itself and inhibits all the other
neurons. To simplify, we will define a transfer function that does the job of
the competitive layer.
a = compet(n)
It works by finding the index i ∗ of the neuron with the largest net input,
and setting its output to 1.

Carlos Borrás
15 - Presentation
4, 2020 Research
8 / 23 G
Competitive Layer
As with the Hamming network, the prototype vectors are stored in rows of
W. The net input n calculates the distance between the input network p
and each prototype i w normalized to L. The angle between them is θi .
 T   2 
1w p L cosθ1
 2 wT p   L2 cosθ2 
n = Wp =  .  =  ..
   
.
 .   .


Sw
Tp L2 cosθS
We can now design the classifier by setting the rows of W to the desired
prototype vectors.
We would like a learning rule that could be used to train the weights in a
competitive network, without knowing the prototype vectors.

Carlos Borrás
15 - Presentation
4, 2020 Research
9 / 23 G
Competitive Layer
For the competitive network, a is only nonzero for winning neuron (i = i ∗ ).

Therefore, we can get the same results using the Kohonen Rule.
i w(q) = (1 − α)i w(q − 1) + αp(q)

And i w(q) =i w(q − 1) for i 6= i ∗ ,
The row of the weight matrix that is closest to the input vector (or has the
largest inner product with the input vector) moves toward the input vector.

Carlos Borrás
15 - Presentation
P., PhD., MSc. (Escuela deNovember
Ingeniería4,
Mecánica
2020 Research
10 / 23 G
Example Kohonen

−0.1961 0.1961 0.9806
p1 = , p2 = , p3 =
0.9806 0.9806 0.1961

0.9806 −0.5812 −0.8137
p4 = ,p = ,p =
−0.1961 5 −0.8137 6 −0.5812
Our competitive network will have three neurons, and therefore it can
classify into three classes.

0.7071 0.7071 −1
1 w = , w = , w =
−0.7071 2 0.7071 3 0

Carlos Borrás
15 - Presentation
Ingeniería4,
Mecánica
2020 Research
11 / 23 G
Example Kohonen
    
0.7071 −0.7071 0
0.1961   
a = compet(Wp2 ) = compet 0.7071 0.7071  = 1
0.9806
−1 0 0
The second neuron weight was closest to p2 , so it won competition

(i ∗ = 2). We now apply Kohonen rule to winning neuron with a learning
rate of α = 0.5.

new old old 0.4516
2w =2 w + α(p2 −2 w =
0.8438

Carlos Borrás
15 - Presentation
Ingeniería4,
Mecánica
2020 Research
12 / 23 G
Problems with Competitive Layers
First, the choice of learning rate force a trade-off between the speed of
learning and the stability of the final weight vectors.
Initial training can be done with a large learning rate for fast learning.
Then the learning rate can be decreased as training progresses, to achieve
stable prototype. However, this doesn’t work if the network needs to
continuously adapt to a new arrangement of input vectors.

Carlos Borrás
15 - Presentation
Ingeniería4,
Mecánica
2020 Research
13 / 23 G
Another problem results when a weight vector forming a prototype of one

cluster may invade the territory of another weight vector.
Eventually one of the right cluster vectors is reclassified by the center

weight vector, leading to losses of some of its vectors.

Carlos Borrás
15 - Presentation
Ingeniería4,
Mecánica
2020 Research
14 / 23 G
Another problems results when an initial weight vector is located so far

from any input vectors that it never wins competition, therefore, the result
is a "dead" neuron.
One solution consists of adding a negative bias to the net input of each
neuron and then decreasing the bias each time the neuron wins.

Carlos Borrás
15 - Presentation
Ingeniería4,
Mecánica
2020 Research
15 / 23 G
Self-Organizing Feature Maps
The SOFM first determines the winning neuron i ∗ using the same
procedure as the competitive layer. Next, the weight vectors for all neuron
within a certain neighborhood of the winning neuron are updated with
Kohonen rule.
i w(q) =i w(q − 1) + α(p(q) −i w(q − 1))

Where neighborhood Ni∗ (d) contains the indices for all of the neurons that
lie within a radius d of the winning neuron i ∗ .
Ni (d) = {j, dij ≤ d}

When a vector p is presented, the weight of the winning neuron and its
neighbors will move toward p. The result is that, after many presentations,
neighboring neurons will have learned vectors similar to each other.

Carlos Borrás
15 - Presentation
Ingeniería4,
Mecánica
2020 Research
16 / 23 G
Kohonen has also suggested rectangular and hexagonal neighborhoods for

efficient implementation.

Carlos Borrás
15 - Presentation
Ingeniería4,
Mecánica
2020 Research
17 / 23 G
Each three element weight vector is represented in the left by a dot in the
sphere. The line connections are done to represent topology. To the right
we have a square region surface of a sphere, where we get the feature map
from.
Each time a vector is presented, the neuron with the closest weight vector
will win the competence. Then, the winning neuron move their weight
vectors closer to the input.

Carlos Borrás
15 - Presentation
Ingeniería4,
Mecánica
2020 Research
18 / 23 G
Weight vectors have two tendencies: first, they spread out over the input
space as more vectors are presented; second, they move toward the weight
vectors of neighboring neurons. Thus we rearrange to classify the input
space.
The input vectors were generate with equal probability from any point in
the input space. Therefore, the neurons classify roughly equal areas of the
input space.

Carlos Borrás
15 - Presentation
Ingeniería4,
Mecánica
2020 Research
19 / 23 G
Maps can also fail to properly fit topology. This usually occurs when two
pats of the net fit the topology of separate parts of the input space, but
the net forms a twist between them.

Carlos Borrás
15 - Presentation
Ingeniería4,
Mecánica
2020 Research
20 / 23 G
Improving Feature Maps

We want to speed up the self-organization process and to make it more
reliable.
One method is to vary the size of the neighborhoods during training.
Initially, the neighborhood size, d, is set large. As training progresses, d is
gradually reduced, until it includes only the winning neuron.
The learning rate can also be varied over time. An initial rate of 1 allows
neurons to quickly learn presented vectors. During training, the learning
rate decreases asymptotically toward 0 and it becomes stable.

Carlos Borrás
15 - Presentation
Ingeniería4,
Mecánica
2020 Research
21 / 23 G
Learning Vector Quantization
In LVQ, each neuron in the first layer is assigned to one neuron in the
second layer. The number of networks in the first layer, S 1 , will therefore
be at least as large as the number of neurons in the second layers, S 2 , and
will usually be larger.
As with competitive network, each neuron in the first layer of the LVQ
network learns a prototype vector, which allows it to classify a region of the
input space.

Carlos Borrás
15 - Presentation
Ingeniería4,
Mecánica
2020 Research
22 / 23 G
Learning Vector Quantization
We will simulate the LVQ network by calculating the distance directly. In

this case, vectors do not need to be normalized.
ni0 = − i w1 − p

In vector form,
 1 
1 w − p
1
 2 w − p 
n1 = −  ..
 

1.
 

S w − p
Output is,
a1 = compet(n1 )

Carlos Borrás
15 - Presentation
Ingeniería4,
Mecánica
2020 Research
23 / 23 G

Competitive Networks: Master Student: Jean Carlo Grandas F. Professor: Carlos Borrás P., PHD., MSC

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Competitive Networks: Master Student: Jean Carlo Grandas F. Professor: Carlos Borrás P., PHD., MSC

Uploaded by

Copyright:

Available Formats

Competitive Networks

Master Student: Jean Carlo Grandas F.

Escuela de Ingeniería Mecánica

Research Group: Dinámica Multifísica, Control y Robótica. DICBoT

Master Student: Jean Carlo Grandas F.Professor: Chapter

Hamming network consist of two layers. The first one performs a

Master Student: Jean Carlo Grandas F.Professor: Chapter

Master Student: Jean Carlo Grandas F.Professor: Chapter

Thus the output is,

Master Student: Jean Carlo Grandas F.Professor: Chapter

Master Student: Jean Carlo Grandas F.Professor: Chapter

Then, the second-layer output is updated according to following recurrence

Master Student: Jean Carlo Grandas F.Professor: Chapter

To illustrate this effect on the outputs,

At each iteration, each neuron’s output will decrease in proportion to the

Master Student: Jean Carlo Grandas F.Professor: Chapter

Master Student: Jean Carlo Grandas F.Professor: Chapter

Master Student: Jean Carlo Grandas F.Professor: Chapter

For the competitive network, a is only nonzero for winning neuron (i = i ∗ ).

i w(q) = (1 − α)i w(q − 1) + αp(q)

Master Student: Jean Carlo Grandas F.Professor: Chapter

Master Student: Jean Carlo Grandas F.Professor: Chapter

The second neuron weight was closest to p2 , so it won competition

Master Student: Jean Carlo Grandas F.Professor: Chapter

Master Student: Jean Carlo Grandas F.Professor: Chapter

Another problem results when a weight vector forming a prototype of one

Eventually one of the right cluster vectors is reclassified by the center

Master Student: Jean Carlo Grandas F.Professor: Chapter

Another problems results when an initial weight vector is located so far

Master Student: Jean Carlo Grandas F.Professor: Chapter

i w(q) =i w(q − 1) + α(p(q) −i w(q − 1))

Ni (d) = {j, dij ≤ d}

Master Student: Jean Carlo Grandas F.Professor: Chapter

Kohonen has also suggested rectangular and hexagonal neighborhoods for

Master Student: Jean Carlo Grandas F.Professor: Chapter

Master Student: Jean Carlo Grandas F.Professor: Chapter

Master Student: Jean Carlo Grandas F.Professor: Chapter

Master Student: Jean Carlo Grandas F.Professor: Chapter

Improving Feature Maps

Master Student: Jean Carlo Grandas F.Professor: Chapter

Master Student: Jean Carlo Grandas F.Professor: Chapter

We will simulate the LVQ network by calculating the distance directly. In

Master Student: Jean Carlo Grandas F.Professor: Chapter

You might also like