Feedback Networks

and Associative Memories

虞台文
大同大學資工所
智慧型多媒體研究室
Content
 Introduction
 Discrete Hopfield NNs
 Continuous Hopfield NNs
 Associative Memories
– Hopfield Memory
– Bidirection Memory
 Feedback Networks
and Associative Memories

Introduction
大同大學資工所
智慧型多媒體研究室
Feedforward/Feedback NNs

 Feedforward NNs
– The connections between units do not form cycles.
– Usually produce a response to an input quickly.
– Most feedforward NNs can be trained using a wide variet
y of efficient algorithms.
 Feedback or recurrent NNs
– There are cycles in the connections.
– In some feedback NNs, each time an input is presented, t
he NN must iterate for a potentially long time before it p
roduces a response.
– Usually more difficult to train than feedforward NNs.
Supervised-Learning NNs
 Feedforward NNs
– Perceptron
– Adaline, Madaline
– Backpropagation (BP)
– Artmap
– Learning Vector Quantization (LVQ)
– Probabilistic Neural Network (PNN)
– General Regression Neural Network (GRNN)
 Feedback or recurrent NNs
– Brain-State-in-a-Box (BSB)
– Fuzzy Congitive Map (FCM)
– Boltzmann Machine (BM)
– Backpropagation through time (BPTT)
Unsupervised-Learning NNs
 Feedforward NNs
– Learning Matrix (LM)
– Sparse Distributed Associative Memory (SDM)
– Fuzzy Associative Memory (FAM)
– Counterprogation (CPN)
 Feedback or Recurrent NNs
– Binary Adaptive Resonance Theory (ART1)
– Analog Adaptive Resonance Theory (ART2, ART2a)
– Discrete Hopfield (DH)
– Continuous Hopfield (CH)
– Discrete Bidirectional Associative Memory (BAM)
– Kohonen Self-organizing Map/Topology-preserving map
(SOM/TPM)
The Hopfield NNs
 In 1982, Hopfield, a Caltech physicist,
mathematically tied together many of the
ideas from previous research.
 A fully connected, symmetrically weighted
network where each node functions both
as input and output node.
 Used for
– Associated memories
– Combinatorial optimization
Associative Memories
 An associative memory is a content-addressable s
tructure that maps a set of input patterns to a se
t of output patterns.
 Two types of associative memory: autoassociative
and heteroassociative.
 Auto-association
– retrieves a previously stored pattern that most closely r
esembles the current pattern.
 Hetero-association
– the retrieved pattern is, in general, different from the i
nput pattern not only in content but possibly also in type
and format.
Associative Memories
Auto-association

A memory
A

Hetero-association

Niagara Waterfall
memory
Optimization Problems
 Associate costs with energy functions in H
opfield Networks
– Need to be in quadratic form

 Hopfield Network finds local, satisfactory

soluions, doesn’t choose solutions from a se
t.

 Local optimums, not global.

 Feedback Networks
and Associative Memories

Discrete Hopfi
eld NNs
大同大學資工所
智慧型多媒體研究室
The Discrete Hopfield NNs

w1n w2n w3n

w13 w23 wn3
w12 w32 wn2
w21 w31 wn1

1 2 3 ... n

I1 I2 I3 In

v1 v2 v3 vn
 w
The Discrete Hopfield NNs
ij = w ij

w1n
wii = 0
w2n w3n
w13 w23 wn3
w12 w32 wn2
w21 w31 wn1

1 2 3 ... n

I1 I2 I3 In

v1 v2 v3 vn
 w
The Discrete Hopfield NNs
ij = w ij

w1n
wii = 0
w2n w3n
w13 w23 wn3
w12 w32 wn2
w21 w31 wn1

1 2 3 ... n

I1 I2 I3 In

v1 v2 v3 vn
State Update Rule w1n w2n w3n
w13 w23 wn3
w12 w32 wn2
w21 w31 wn1

 Asynchronous mode 11 22 33 ... nn

I1 I2 I3 In

 Update rule v1 v2 v3 vn

n
H i (t  1)   wij v j (t )  I i
Sta b l e ?
j 1
j i

 1 H i (t  1)  0
vi (t  1)  sgn  H i (t  1)   
1 H i (t  1)  0
 n
H i (t  1)   wij v j (t )  I i
j 1
j i

 1 H i (t  1)  0

Energy Function
vi (t  1)  
1 H i (t  1)  0

w1n w2n w3n

w13 w23 wn3
w12 w32 wn2
w21 w31 wn1
n n n ...
 w v v   I v
11 22 33 nn
E 1
2 ij i j i i
I1 I2 I3 In
i 1 j 1 i 1

Fact:E is lower bounded v1 v2 v3 vn

(upper bounded).

If E is monotonically decreasing
(increasing), the system is
stable.
 n
H i (t  1)   wij v j (t )  I i
j 1
j i

 1 H i (t  1)  0

The Proof
vi (t  1)  
1 H i (t  1)  0

w1n w2n w3n

w13 w23 wn3
w12 w32 wn2
w21 w31 wn1
n n n ...
 w v v   I v
11 22 33 nn
E 1
2 ij i j i i
I1 I2 I3 In
i 1 j 1 i 1

Suppose that at time t + 1, the kth v1 v2 v3 vn

neuron is selected for update.
n n n n
E (t )   1
2  w v (t )v (t )   I v (t )  w v (t )v (t )  I v (t )
i 1 j 1
ij i j
i 1
i i
i 1
ki k i k k

ik jk ik

n n n n
E (t  1)   1
2  w v (t  1)v (t  1)   I v (t  1)  w v (t  1)v (t  1)  I v (t  1)
i 1 j 1
ij i j
i 1
i i
i 1
ki k i k k

ik jk ik

 n
H i (t  1)   wij v j (t )  I i
j 1
j i

 1 H i (t  1)  0

The Proof
vi (t  1)  
1 H i (t  1)  0

w1n w2n w3n

w13 w23 wn3
w12 w32 wn2
w21 w31 wn1
n n n ...
 w v v   I v
11 22 33 nn
E 1
2 ij i j i i
I1 I2 I3 In
i 1 j 1 i 1

Their values are not

Suppose that at time t + 1, the kth v1 v2 at time
v3 t + vn
changed
neuron is selected for update. 1.
n n n n
E (t )   1
2  w v (t )v (t )   I v (t )  w v (t )v (t )  I v (t )
i 1 j 1
ij i j
i 1
i i
i 1
ki k i k k

ik jk ik

n n n n
E (t  1)   1
2  w v (t  1)v (t  1)   I v (t  1)  w v (t  1)v (t  1)  I v (t  1)
i 1 j 1
ij i j
i 1
i i
i 1
ki k i k k

ik jk ik

 n
H i (t  1)   wij v j (t )  I i
j 1
j i

 1 H i (t  1)  0

The Proof
vi (t  1)  
1 H i (t  1)  0

n n
E  E (t  1)  E (t )   wki vk (t  1)vi (t )  I k vk (t  1)   wki vk (t )vi (t )  I k vk (t )
i 1 i 1

 n 
    wki vi (t )  I k  [vk (t  1)  vk (t )]
 i 1 
  H k (t )[vk (t  1)  vk (t )]

n n n n
E (t )   1
2  w v (t )v (t )   I v (t )  w v (t )v (t )  I v (t )
i 1 j 1
ij i j
i 1
i i
i 1
ki k i k k

ik jk ik

n n n n
E (t  1)   1
2  w v (t  1)v (t  1)   I v (t  1)  w v (t  1)v (t  1)  I v (t  1)
i 1 j 1
ij i j
i 1
i i
i 1
ki k i k k

ik jk ik

 n
H i (t  1)   wij v j (t )  I i
j 1
j i

 1 H i (t  1)  0

The Proof
vi (t  1)  
1 H i (t  1)  0

n n
E  E (t  1)  E (t )   wki vk (t  1)vi (t )  I k vk (t  1)   wki vk (t )vi (t )  I k vk (t )
i 1 i 1

 n 
    wki vi (t )  I k  [vk (t  1)  vk (t )]
 i 1 
  H k (t )[vk (t  1)  vk (t )]

vk(t) Hk(t) vk(t+1) E

e
E
1 0 1 0
St a b l
1 <0 1 <0
1 0 1 <0
1 <0 1 0
 Feedback Networks
and Associative Memories

Continuous H
opfield NNs
大同大學資工所
智慧型多媒體研究室
The Neuron of Continuous Hopfield NNs

wi1 Ii vi =a(ui)
v1 1
wi2
v2 .. ui

vn
. 1

win
ui ui =a1(vi)

Ci gi
a (ui ) 1
1 vi

vi vi
The Dynamics
wi1 Ii dui n

(v1  ui ) wi1 Ci   wij (v j  ui )  gi ui  I i

v1 dt j 1
wi2 j i

v2 .. (v2  ui ) wi 2
dui n
 n 
  wij v j    wij  gi ui  I i
vn
. (vn  ui ) win
Ci
dt j 1
j i
 j 1
 j i


win
ui
C dui
i dt
 gi ui
Gi
Ci gi
a (ui )
dui n
Ci   wij v j  Gi ui  I i
dt j 1
vi vi j i
 dui n
Ci   wij v j  Gi ui  Ii
dt j 1
j i

The Continuous Hopfield NNs

I1 I2 I3 In
w1n
w3n
w2n
wn3
w13 w23
w12 w32
wn2
w31
w21 wn1
u1 u2 u3 un
C1 g1 C2 g2 C3 g3 Cn gn
...
v1 v2 v3
. . .v
n

v1 v2 v3 vn
 dui n
Ci   wij v j  Gi ui  Ii
dt j 1
j i

The Continuous Hopfield NNs

du1 n
C1   w1 j v j  G1u1  I1
I1 I2 I3 dt j 1 In
j i
w1n
w3n du2 n
w2n C2   w2 j v j  G2u2  I 2
dt j 1 wn3
w13 w23 j i
w12 w32
w

e?
w31 du3 n n2

Sta b
u1l C1 g1
u2
w21

C2 g2
u3
C3
C3
dt
g3
  ww3 jn1v j  G3u3  I 3
j 1u
j i n
Cn gn
...
v1 v2 v3
. . .v
n
dun n
Cn   wnj v j  Gnun  I n
dt j 1
j i
v1 v2 v3 vn
Equilibrium Points
 Consider the autonomous system:
y 1  f1 (y )
y 2  f 2 (y )

y (t )  f (y )
y n  f n (y )

 Equilibrium Points Satisfy

0  f (y*)
 Call E(y) as energy function.

Lyapunov Theorem

y (t )  f (y )
The system is asymptotically stable if the
following holds:

There exists a positive-de dE (y )

finite function E(y) s.t.
0
dt
 1 n n n
1 n vi
E    wij vi v j   I i vi   Gi  a 1 (v)dv
2 i 1 j 1 i 1  i 1 0
j n

Lyapunov Energy Function

I1 I2 I3 In
w1n
w3n
w2n
wn3
w13 w23
w12 w32
wn2
w31
w21 wn1
u1 u2 u3 un
C1 g1 C2 g2 C3 g3 Cn gn
...
v1 v2 v3
. . .v
n

v1 v2 v3 vn
 1 n n n
1 n vi
E    wij vi v j   I i vi   Gi  ai1 (v)dv
2 i 1 j 1 i 1  i 1 0
jn

Lyapunov Energy Function

u=a1(v) v
vi =a(ui) da 1 (v)
0 
0
a 1 ( x)dx
1 dv
I1 I2 I3 In
1
uwi 1n 1
w3n
v
1
w2n

w13 w23 1 wn3 1 v

w12 w32
wn2
w31
w21 wn1
1
a (v)  u u1 u2 u3 un
C1 g1 C2 g2 C3 g3 Cn gn
...
v1 v2 v3
. . .v
n

v1 v2 v3 vn
 dui n
Dynamics Ci   wij v j  Gi ui  I i
dt j 1
j i

Stability of Continuous Hopfield NNs

1 n n n
1 n vi
E    wij vi v j   I i vi   Gi  ai1 (v)dv
2 i 1 j 1 i 1  i 1 0
j n

 n 
dE n
dE dvi n
  dvi
    wij v j  Gi ui  I i
dt i 1 dvi dt  j 1
i 1 
 dt
 j i 
n
dui dvi
  Ci 1 1 1
i 1 dt dt  u i  a i ( vi ) ui   ai (vi )
2
1 n dai1 (vi )  dvi 
   Ci   dui 1 dai1 (vi ) dvi
 i 1 dvi  dt  
dt  dvi dt
 dui n
Dynamics Ci   wij v j  Gi ui  I i
dt j 1
j i

Stability of Continuous Hopfield NNs

1 n n n
1 n vi
E    wij vi v j   I i vi   Gi  ai1 (v)dv
2 i 1 j 1 i 1  i 1 0
j n

dE 1 n 1
da (vi )  dvi 
2
da 1 (v)
   Ci i 0
dt  i 1 
dvi  dt   dv
u=a1(v)
>0

dE 1

0 1 v

dt
Stability of Continuous Hopfield NNs
du1 n
C1   w1 j v j  G1u1  I1
I1 I2 I3 dt j 1 In
j i
w1n
w3n du2 n
w2n C2   w2 j v j  G2u2  I 2
dt j 1 wn3
w13 w23 j i
w12 w32
w

e
w31 du3 n n2

Sta b
u1l C1 g1
u2
w21

C2 g2
u3
C3
C3
dt
g3
  ww3 jn1v j  G3u3  I 3
j 1u
j i n
Cn gn
...
v1 v2 v3
. . .v
n
dun n
Cn   wnj v j  Gnun  I n
dt j 1
j i
v1 v2 v3 vn
 Feedback Networks
and Associative Memories

Associative
Memories
大同大學資工所
智慧型多媒體研究室
Associative Memories
 Also named content-addressable memory.

 Autoassociative Memory
– Hopfield Memory

 Heteroassociative Memory
– Bidirection Associative Memory (BAM)
Associative Memories
Stored Patterns
(x1 , y1 ) i i
2
(x , y )2 x y Autoassociative

 i i
p
(x , y )p x y Heteroassociative
i n
x R
yi  Rm
 Feedback Networks
and Associative Memories

Associative Memories
 Hopfield Memory
 Bidirection Memory

大同大學資工所
智慧型多媒體研究室
Hopfield Memory

Fully connected

14,400 weights
1210
Neurons
Example

Stored Memory
Patterns Association
Example

Stored Memory
Patterns Association
Example t o r e Pa t t e r n s ?
How t o S

Stored Memory
Patterns Association
The Storage Algorithm
Suppose the set of stored pattern is of dimension n.

 w11 w12  w1n 

=?
w w22 
 w2 n 
W  21

    
 
 wn1 wn 2  wnn 
The Storage Algorithm
k k k k T
x  ( x , x , , x )
1 2 n k  1, 2,  p.
k
x  {1, 1} i  1, 2, n.
i

 p k k
p
 xi x j i j
W   x k (x k )T  pI wij   k 1
k 1 0 i j

  p k k
p
 xi x j i j
W   x ( x )  pI
k k T
wij   k 1
k 1 0 i j
Analysis


Suppose that x  xi.

p
 
Wx    x (x )  pI  x
k k T

 k 1 
i i i
 nx  px  (n  p)x
  p k k
p
 xi x j i j
W   x ( x )  pI
k k T
wij   k 1

Example
k 1 0 i j


 1  1  1 1
1 T
x  (1, 1, 1,1) 1 1 T
 1 1 1  1
x (x )   
 1 1 1  1
x 2  (1,1, 1,1)T 
 1  1  1 1


 1  1 1  1
 1 1  1 1
x (x )  
2 2 T 
 0 2 0 0   1  1 1  1
 2 0 0 0  
  1 1  1 1


W 
 2 2 0 0
 0 0 0 2   2 
  x (x )  x (x )  
1 1 T 2 2 T 2 0 0 
 0 0 2 0   0 0 2  2
 
 0 0 2 2
Example
1 T 2 2
x  (1, 1, 1,1) 1 2 3 4

x 2  (1,1, 1,1)T
n n n
E (x)   12  wij xi x j   I i xi
i 1 j 1 i 1
 0 2 0 0  n n

 2 0 0 0    12  wij xi x j
W  i 1 j 1

 0 0 0 2    12 xT Wx
 
 0 0 2 0   2( x1 x2  x3 x4 )
Example
1 T 2 2
x  (1, 1, 1,1) 1 2 3 4

x 2  (1,1, 1,1)T 1 1 1 1
E=4
1 1 1 1

E (x)  2( x1 x2  x3 x4 ) E=0
1 1 1 1

E=4
Stable
Example
1 T 2 2
x  (1, 1, 1,1) 1 2 3 4

x 2  (1,1, 1,1)T 1 1 1 1
E=4
1 1 1 1

E (x)  2( x1 x2  x3 x4 ) E=0
1 1 1 1

E=4
Stable
Problems of Hopfield Memory

 Complement Memories
 Spurious stable states
 Capacity
Capacity of Hopfield Memory

 The number of storable patterns w.r.t. the

size of the network.

 Study methods:
– Experiments
– Probability
– Information theory
– Radius of attraction ()
Capacity of Hopfield Memory

 The number of storable patterns w.r.t. the

size of the network.

Hopfield (1982) demonstrated that the maximum number of patterns tha

t can be stored in the Hopfield model of n nodes before the error in the
retrieved pattern becomes severe is around 0.15n. 

The memory capacity of the Hopfield model can be increased as shown b

y Andrecut (1972).
Radius of attraction (0    1/2)

n
(1  2  ) 2 n
c
4 ln n
False memories
 Feedback Networks
and Associative Memories

Associative Memories
 Hopfield Memory
 Bidirection Memory

大同大學資工所
智慧型多媒體研究室
 Bidirection Memory
y1 y2 yn

y (t  1)  a  Wx(t )  ...
Y Layer
y (t )

Backward
.
Forward

Pass
W=[wij]nm
Pass

x(t ) .. x(t   1)  a  WT y (t ) 
X Layer
x1 x2 xm
 Stored Patterns
1 1
(x , y )
Bidirection Memory (x , y ) 2 2

y1 y2 yn

(x p , y p )
y (t  1)  a  Wx(t )  ...
Y Layer
y (t )

?
Backward
.
Forward

Pass
W=[wij]nm
Pass

x(t ) .. x(t   1)  a  WT y (t ) 
X Layer
x1 x2 xm
The Storage Algorithm
k T
Stored x  ( x1 , x2 , , xm ) xi  {1,1}
Patterns k
y  ( y1 , y2 , , yn ) T
yi  {1,1}
(x1 , y1 ) p
2
(x , y ) 2 W   y k (x k )T
k 1

p
p
(x , y ) p
wij   yik x kj
k 1
 p
W   y k (x k )T
Analysis k 1

Suppose xk’ is one of the stored vector.

 p
k T k 
y  a  Wx   a   y (x ) x 
k k

 k 1 
 
 a  my y
p
 k T k  k k
 a my   y (x ) x
k k
 k 1

 k k ' 
0

Energy Function

T T T
E (x, y )   x W y  y Wx
1
2
1
2
T
  y Wx
 Bidirection Memory
Energy Function:

T T T
E (x, y )   x W y  y Wx
1
2
1
2
T
 y Wx