Pattern Association

Klinkhachorn:CpE320

Associative Memory Neural Nets

An associate network is a neural network
with essentially a single functional layer that
associates one set of vectors with another set
of vectors
The weights are determined in such a way
that the net can store a set of pattern
associations
Each association is an input-output vector
pair, s:t

Klinkhachorn:CpE320

Associative Memory (Cont.)

If each vector t is the same as the vector s
which it is associated, then the net is called an

autoassociative memory
If the ts are different from the ss, the net is
called a heteroassociative memory
The net not only learns the specific pattern
pairs that were used for training, but also is
able to recall the desired response pattern
when given an input stimulus that is similar
but not identical, to the training input.
Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

Klinkhachorn:CpE320

Associated Pattern Pairs

Klinkhachorn:CpE320

Partial Data Presented

Who?
Astro

Rosie

Look like?

Klinkhachorn:CpE320

Noisy Data Presented

Klinkhachorn:CpE320

Noisy Data Presented

Klinkhachorn:CpE320

Classification of ANN
Paradigms

Klinkhachorn:CpE320

Architecture of an Associative memory

Neural Net

Feedforward net: information flows from the

input units to the output units
Example: Linear associator Network
Recurrent (iterative): there are connections among
the units that form closed loop
Example: Hopfield, BAM

Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

Klinkhachorn:CpE320

Hebb Rule for Pattern Association

Simplest and most common method of
determining the weights for an associative
memory neural net
Can be used with patterns that are represented as
either binary or bipolar vectors
x1
w1
xn

e = wixi

y = 1 if e 0
= 0 if e < 0

wn

Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

Klinkhachorn:CpE320

Algorithm
Step 1: Initialize all weights (i=1,.,n; j = 1..,m)
wij = 0
Step 2: For each input training-target output vector pair
s:t, do Steps 3-5.
Step 3: Set activations for input units to current
training input (i = 1,.,n):
xi=si
Step 4: Set activations for output units to current
target output (j = 1,.,m):
yj = tj
Step 5: Adjust the weights( I=1,n; j=1,.,m):
wij(new) = wij(old) +xiyi
Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

Klinkhachorn:CpE320

Weights Calculation: Outer Product

Let s-t be the input vector-output vector pairs
The outer product of two vectors
s = (s1,..,si,.,sn)
and
t = (t1,...,tj,.,tm)
Is the matrix product of the n x 1 matrix S = sT and
the 1 x m matrix T = t:

Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

Klinkhachorn:CpE320

Weights Calculation: Outer Product (Cont)

To store a set of associations s(p) : t(p), p=1,.,P, where
s(p) = (s1(p),.,si(p),.,sn(p))
and
t(p) = (t1(p),.,tj(p),.,tm(p))
The weight matrix W = {wij} is given by
P

wij = si(p)tj(p)
p =1

This is the sum of the outer product matrices required to

store each association separately.
P

In general,

W = sT(p)t(p) = sTt
p =1

Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

Klinkhachorn:CpE320

Perfect recall VS cross talk

Hebb rule:
If the input vectors are uncorrelated (orthogonal), the
Hebb rule will produce the correct weights, and the
response of the net when tested with one of the training
vectors will be perfect recall
If the input vectors are not orthogonal, the response
will include a portion of each of their target values.
This is commonly called cross talk

Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

Klinkhachorn:CpE320

Hebb rule: an example

Suppose a Heteroassociative net is to be trained to
store the following mapping from input row vectors
s = (s1,s2,s3,s4) to output row vectors t = (t1,t2):

Pattern
1st
s
2nd s
3rd s
4th s

s1,s2,s3,s4
(1, 0, 0, 0)
(1, 1, 0, 0)
(0, 0, 0, 1)
(0, 0, 1, 1)

Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

t1,t2
(1, 0)
(1, 0)
(0, 1)
(0, 1)

Klinkhachorn:CpE320

Hebb rule: an example

x1
x2
x3
x4

w11
w12

w21
w22
w31
w32
w41
w42

x1
x1

Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

Klinkhachorn:CpE320

Hebb rule: an example

Weights calculation: Outer Product
4

W = sT(p)t(p) = sTt
p =1

1
0
W=
0
0

1
1
0
0

0
0
0
1

0 1
0 1
.
1 0
1 0

0 2
0 1
=
1 0
1 0

Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

0 w11
0 w21
=
1 w31
2 w41

w12
w22
w32
w42

Klinkhachorn:CpE320

Hebb rule: an example

Testing the net
for x = (1,0,0,0)

2
1
xW = (1,0,0,0)
0
0

0
0
=
(2,0)

>
(1,0)
1
2

Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

Klinkhachorn:CpE320

Hebb rule: an example

Testing the net - input similar to the training input
for x = (0,1,0,0) differs from the training vector (1,1,0,0)
only in the first component.

2
1
xW = (0,1,0,0)
0
0

0
0
=
(1,0)

>
(1,0)
1
2

Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

Klinkhachorn:CpE320

Hebb rule: an example

Testing the net - input not similar to the training input
for x = (0,1,1,0) differs from the training input patterns in
at least two components.

2
1
xW = (0,1,1,0)
0
0

0
0
=
(1,1)

>
(1,1)
1
2

The two mistakes in the input pattern make it

impossible for the net to recognize it!
Laurene Fausett, Fundamentals of Neural Networks, Prentice Hall

Klinkhachorn:CpE320

Iterative Autoassociative Net

Klinkhachorn:CpE320

Hopfield Nets (Recurrent)

Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Klinkhachorn:CpE320

John Hopfield
Computational neurobiology /
biophysics

While the brain is totally unlike current digital computers, much of what it does can be described as computation.
Associative memory, logic and inference, recognizing an odor or a chess position, parsing the world into objects, and
generating appropriate sequences of locomotor muscle commands are all describable as computation. My current
research focuses on the theory of how the neural circuits of the brain produce such powerful and complex
computations. Olfaction is one of the oldest and simplest senses, and much of my recent work derives from
considerations of the olfactory system. One tends to think of olfaction as "identifying a known odor," but in highly
olfactory animals, the problems solved are much more complicated. Many animals use olfaction as a remote sense, to
understand the environment around them in terms of identifying both where and what objects are remotely located.
This involves at least coordinating when something is smelled with the wind direction, and untangling a weak

signal from an odor object from a background of other odors simultaneously present. The homing of
pigeons or the ability of a slug to find favorite foods are examples of such remote sensing.

http://www.molbio.princeton.edu/faculty/hopfield.html

Klinkhachorn:CpE320

Binary Hopfield Processing Unit

x1
x2

wj1

ej=xiwji

wj2
wj3

yj = +1 if ej0
= 0 if ej<0

x3

i
ij

wjn

xn
Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Klinkhachorn:CpE320

Hopfield Architecture

Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Klinkhachorn:CpE320

Hopfield Net

Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Klinkhachorn:CpE320

Hopfield Net Algorithm

Step1: Compute the connection weights,
Let
m be the total # of associated patterns, i.e.
X = (x1,x2,xp,..,xm)
and for each pattern p with n inputs
xp = (xp1,xp2,,xpi,..xpn)
Then, the connection weight from node i to node j,
m

wij = (2xpi-1)(2xpj-1) . If xp =(0,1), and wii = 0

p=1

or
m

wij = xpixpj

. If xp =(-1,1), and wii = 0

W = XTX

. If xp =(-1,1), and wii = 0

p=1

or

Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Klinkhachorn:CpE320

Hopfield Net Algorithm (Cont)

Step2: Initialize the network with unknown input pattern,
x = (x1, x2,.. , xn), by assigning output
yi(0) = xi, for i= 1 to n
where yi(0) is the output of node i at time 0
Step 3: Iterate (update outputs) until convergence
n

yj(t+1) = F[ wijyij(t)], j = 1 to n
i=1

where F[e] = +1. if e0, or

= -1(0). if e<0

Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Klinkhachorn:CpE320

Hopfield Net: Updating Procedure

Random Update
Allows all units to have the same average update
Sequence Update
Repeat the sequence until a stable state is attained
Most other neural network paradiagms have a layer of
processing units updated at the same time (or nearly the
same time).
Hopfield, however, suggested that random update of the
neuron has advantages both in implementation (each unit
can generate its next update time) and in function
(sequential updating can restrict the output states of the
network in cases in which different stable states are
equiprobable)
Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Klinkhachorn:CpE320

Hopfield Net: Updating Procedure

Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Klinkhachorn:CpE320

Hopfield Net: Analysis

Convergence**
Each state of the Hopfield network has an associated
"Energy" value,
E=

1
w ji x j xi

2 j i, j j

Successive updating the network provides a convergence

procedure thus the energy of the overall network gets
smaller.
Consider that unit j is the next processing unit to be
updated,
1
1
E j = w ji x j xi = x j wji x i
2 i, i j
2 i ,i j

Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Klinkhachorn:CpE320

Hopfield Net: Analysis

When unit j is updated,
1
E j = Einew E jold = x j w ji x i
2
where
x j = x j new x jold

If xj changes from 0 to 1, then

x j = 1
and

ji

xi 0

Thus,
E j 0
Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Klinkhachorn:CpE320

Hopfield Net: Analysis

If xj changes from 1 to 0, then
x j = 1
and

ji

xi < 0

Again,
E j < 0

The network guaranteed to converge, when E

taking on lower and lower values until it reaches
a steady state.
Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Klinkhachorn:CpE320

Hopfield Net: Recalling

T+10% Noise -->T

T +20% Noise --> O

Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Klinkhachorn:CpE320

Hopfield Net: Application

A binary Hopfield net can be used to determine
whether an input vector is a known vector (i.e.,
one that was stored in the net) or an unknown
vector.
If the input vector is an unknown vector, the
activation vectors produced as the net iterates will
converge to an activation vector that is not one of
the stored patterns; such a pattern is called a
spurious stable state.

Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Klinkhachorn:CpE320

Hopfield Net: Storage Capacity

Hopfield found experimentally that the number of
binary patterns that can be stored and recalled in a net
with reasonable accuracy, is approximately
P 0.15 n, where n is the number of neurons in the net
A detailed theoretical analysis showed
P n/2 log2n

Judith Dayhoff, Neural Network Architectures: An Introduction, Van Nostrand Reinhold

Klinkhachorn:CpE320