Pattern association

Associative memories provide an approach to the

computer engineering problem of storing and retrieving data
based on content rather than storage address . informn storage
NN is distributed throughout the weights , a pattern does not
have a storage address in the same sense that it would if it were
. stored in a traditional computer

Associative memory NNs are single layer nets is which the

weights are determined in such way that the net can store a set
of pattern associations . each association is an inputoutput
vector pair s.t .if each vector t is the same as the vector s with
which it is associated , then the net is called auto associative
memory . if the ts are different from the ss , the net is called
. hetero associative memory

In each of cases , the net not only learns the specific

pattern pairs that were used for training but also is able to
recall the desired response pattern were given an input stimulus
.that is similar but not identical to training input

Before computation an associative memory NN , the

original patterns must be converted to an appropriate
representation for computation . however , different
representations of the same pattern may not be equally
powerful or efficient . In a simple problem , the pattern may be
a binary vector or bipolar vector . the input and the output
values can be real or binary . The architecture of anassociative
. memory NN may be feedforword or recurrent (iterative)

In a feedforword net , inform n flows from the input units

to the output units . In a recurrent net ,there are connections
. among the units that form closed loops
A key question for all associative net is how many patterns ( or
pattern pairs ) can be stored or learned before the net starts to
. forget patterns it has learn previously

. A no. of factors influence how much the net can be learn

. the complexity of patterns (the no. of components) -
the similarity of the input patterns that are associated with -
significantly different response patterns

Heteroassociative memory NN

The associative NN can store a set of pattern associations .

each associations is apair of vectors ( s(p) , t(p) ) with p = 1,2,
. p
Each vector s(p) has n components (n-tuple) and each t(p) is an
m-tuple . the weights may be found using the hebb rule or the
. delta rule

In general , a bipolar represent n of the patterns is

computationally preferable to binary representation . One
advantage of bipolar representation n is that gives a simple way
of exprening two different levels of noise that may be
-: comprised in an input pattern
mining data or a response of unsure denoted by 0 -1
mistakes where a pattern contains a response of yes when -2
the correct response should be no and vise versa , a response
of yes would be represented by +1,no by -1

The weight matrix may be computed using the Hebb rule

or outer products.To store a set of binary vector pairs s(p):t(p)
for p=1,2,..p , can use a weight matrix formed from the
corresponding bipolar vectors where matrix W={wij} is given
by

wij= (2si(p)-1) (2tj(p)-1)

p

2
 However if the vector pairs are bipolar the matrix W={wij} is
given by

wij= si(p) tj(p) or W= sT(p) t(p)

p p

i.e the weight matrix for the set of patterns is the sum of the
weight matrices to store each pattern pair separately

N.B The outer product of a vector pair is simply the matrix

product of the training vector (column vector written as n x 1
matrix) and the target vector (row vector 1 x m matrix)

:Algorithm
.Step0: initialize wts using either the Hebb rule or the Delta rule
.Step1: For each input vector do step 2-4
Step2: Set activations for input layer units equal to the current vector
xi = si
Step3: Compute the net input to the output units
y-inj = xi wi j
i

Step4: Determine the activation of the output units for bipolar targest
if y inj > 0 1
yj = 0 if y- inj = 0
if y inj < 0 1-
The output vector y gives the pattern associated with the input vector x .The hetero
.associative memory is not iterative
If the responses of the net are binary , a suitable activation fn is given by
if x > 0 1
= F(x)
if x <= 0 0

3
 An example :- (Heteroassociatve net using the Hebb rule )
Suppose the net is to be trained to store the following mapping from input bipolar
vectors S= (S1, S2, S3, S4)
to output vectors with two components t = (t1,t2)
where

S (1) = (1, -1, -1, -1) t (1) = (1, -1)

S (2) = (1, 1, -1, -1) t (2) = (1, -1)
S (3) = (-1, -1, -1, 1) t (3) = (-1, 1)
S (4) = (-1, -1, 1, 1) t (4) = (-1, 1)

The weight matrix is

w = si () tj ( )
p

Hence

4- 4
W = 2 -2
2 2-
4 4-

Where the weight matrix to store the first potties pais is given by the outer product of the
vectors s = (1, -1, -1, -1) and t = (1, -1), the weight matrix is
1 -1 1
1 1- = 1- 1 1-
1 1- 1-
1 1- 1-

4
Similarly to store the 2nd, 3rd, 4th pair. The weight matrix to store all four pattem
.pairs is sum of the weight matrices to store each pattem pais separately, i.e

4- 4 1- 1 1- 1 1- 1 1- 1
2- 2 = 1- 1 + 1- 1 + 1- 1 + 1 1-
2 2- 1 1- 1- 1 1 1- 1 1-
4 4- 1 1- 1 1- 1 1- 1 1-

Testing the net using the training input, note that the net input to any particular output unit
is the dot product of the input vector (row) with the column of the weight matrix that has
the weights for the output unit is question . The row vector with all of the net puts is the
Product of the input vector and the at matrix. i.e. the entire is represented as
X w=(y.in1 , y.in2 )
Hence using the fist input vector
1st ( 1, -1, -1, -1 ) .W = ( 8, -8 ) ( 1, -1 )
2nd ( 1, -1, -1, -1 ) . W = (12, - 12) ( 1, -1 )
3rd ( -1, -1,-1, 1 ) . W = (-8, 8) ( -1, 1 )
4th ( -1, -1, 1, 1 ) . W = ( -12, 12 ) ( -1, 1 )

.Using the activation fn. for bipolar targets i.e

if x > 0 1
F(x) = 0 if x =0

5
 if x < 0 1-
The above results show that the correct response has been obtained for all of the training
.patterns
However, using an input vector x = (-1, 1, -1, -1) which is similar to the training vector s =
(1, 1, -1, -1) (the 2nd training vector) and differs only in the first component
Hence (-1, 1, -1, -1). W = (4, -4) (1, -1)
i.e. the net associates a known output pattern with this input
Testing the net with the pattern x = (-1, 1, 1, -1) which differs from each of the training
patterns in at least two components
W = (0, 0) (0, 0) .(1- ,1 ,1 ,1-)
i.e. the output is not one of the outputs with which the net was training i.e. the net does
.not recognize the pattern

Auto associative Net

In this net the training is often called storing the vectors which may be binary or
bipolar. A stored vector can be retrieved from distorted or noisy input if the input is
.sufficiently similar to it
The performance of the net is judged by its ability to reproduce a stored pattern from
.noisy input
Often, the weights on the diagonal (wts that connect on input pattern component to the
.corresponding component in the output pattern) are set to zero
Setting there wts to zero may improve the net's ability to generalize

.
Usually the weights are set using Hebb rule, the X1 y1
.outer product i.e

6
p
Xi Yi
W = s t (p) s (p)
P=1
The auto associative net can be used to determine
Xn Yn
"Whether an input vector is "known" or "unknown
: The procedure is. ( i.e. not stored in the net)
Step0: Set the weights using Hebb rule
Step1: For each testing input vector, do step 2-4
Step2: Set activations of the input units equal to the input vector
Step3: Compute the net in put to each output unit

y_inj = xi xij

.Step 4: Apply activation fn

if y.inj > 0 1
= Yj = f (Y.inj)
if y.inj 0 1-
-:Example
An autoasociative net to store one vector (1, 1, 1, -1)
Step o: compute the weights for the vector (1, 1, 1, -1)
1- 1 1 1
1- 1 1 1
W = 1 1 1 -1
1 1- 1- 1-

Step 1: For testing the input vector

Step 2: X = (1, 1, 1, -1)
Step 3: Y_in = (4, 4, 4, -4)
Step 4: Y = f (4, 4, 4, -4) = (1, 1, 1, -1)

7
 .i.e. The input vector is recognized
N.B. using the net is simply
W . ( ,1- ,1 ,1 ,1 )
Similarly it can be show that the net recognizes the Vectors (-1, 1, 1, -1, )
and ( 1, 1, 1, -1, ) ( ,1- ,1- ,1 ,1 ) , ( ,1- ,1 ,1- ,1 )
.i.e. the net recognizes vector with one mistake in the input vector
.The net can be shown, also to recognize the vector formed when one component is mining i.e
and ( 1, 1, 1, 0 ) ( 1 ,0 ,1 ,1 ) , ( 1- ,1 ,0 ,1) , ( ,1- ,1 ,1 ,0 )

Again , the net can recognize input vectors with two

, mining entres much as : ( 0, 0, 1, -1 ) , ( 0, 1, 0, -1 )
( 0 ,0 ,1 ,1) ( 0 ,1 ,0 ,1) , ( 1- ,0 ,0 ,1) , ( 0 ,1 ,1 ,0)

However , it can be shown that the net does not recognize input vector with two mistakes in
the input vector much as (-1, -1, 1, -1 )

. in general , a net is more tolerant of mining data than it is of mistakes in the data
.it in fairly cornmor for an auto as ociatuie net to have its diagonal terms set to zero . e.g

1- 1 1 0
W0 = 1 0 1 -1
1- 0 1 1
0 1- 1- 1-
.Using this net, it still does not recognize as input vector
Such as (-1,-1.1,-1) with two mustaches.

Stage capacity:
This is numbers of vectors (or pattern pairs) that can stored in the net be in the net
before the net begins to forget.

8
More than one vector can be stored in an auto associative net by adding the weight
matrices for each vector together.
The capacity of an auto associative depends on the net of components the stored
vectors have and
The relationships among the stored vectors: more vectors can be stored if they are
mutually orthogonal.
An auto associative net with four nodes can store three orthogonal vectors (to each
of the other two) However, the wt metrics for four mutually orthogonal vectors is
always singular ( all the elements are zero)

Iterative auto associative net:

Can make the input &output units be the same to obtain a recusant auto associative
net i.e
x1.w =x2 iterate
x2.w = x3 and so on.

Y Y Y Y
1 2 i n

9
10