2 views

Uploaded by Marius_2010

...

- 22trans3
- nervous system powerpoint
- Outcome Gmfm
- Prelim General Psychology
- music
- Bug Algorithms
- Case Study on Pre Eclampsia
- Transport Model
- Germany.pdf
- neuro assessment
- Shortened Halstead Category Test
- Inteliigence is It in Brain or Heart
- Stages
- r_magnus_1926.pdf
- SCI Seminar
- BR 09_Siva_S P Robbins
- senior research
- Project Presentation
- 1. Connected Learning
- Information and Communications Technology Handouts

You are on page 1of 41

Neural Networks

Course 11: Hopfield nets

Overview

Energy in a Hopfield net

Training Hopfield net

Hopfield problems (Spurious minima)

Capacity of Hopfield nets

What is a Hopfield Net

What is a Hopfield net?

popularized by John Hopfield in 1982. It works by first remembering some

binary patterns and then returning the one that is most similar to the input

Due to the way it works, many scientists considered that the Hopfield

network could explain the how brain memories are formed and retrieved

What is a Hopfield net?

binary threshold neurons, where each unit is

symmetrically connected with all the other

units (except itself).

also as output

binary state, that is usually 0 or 1 (can also

be -1 or 1)

What is a Hopfield net?

autoassociative, meaning that it learns to

map a noisy input to a learned pattern

memory only needs some parts to retrieve a

stored item.

than you only need a subset of the bits to

retrieve it

Energy of a Hopfield net

Energy of a Hopfield Net

and analyze, but Hopfield realized that if the

weights are symmetric, the learning stabilizes

at one point.

energy based model, since learning is

directed by an energy function

defined properly, the binary decision rule

causes the network to go down in energy

and stabilize

Energy of a Hopfield Net

Global Energy:

The global energy is the sum of many contributions, computed locally

=

<

where

=

=

So, each unit affects the global energy. The way it affects it, can be computed locally:

= = 0 = 1 = +

Energy of a Hopfield Net

Start from a random state -4

Update units in random order, one at a

time, to the state that gives the lowest

global energy 3 2 3 3

Stop when there is no unit that

changes its output

-1 -1

Energy of a Hopfield Net

Start from a random state -4

1 0

Update units in random order, one at a

time, to the state that gives the lowest

global energy 3 2 3 3

Stop when there is no unit that

changes its output 1 0 0

-1 -1

-E=goodness=3

Energy of a Hopfield Net

Start from a random state -4

1 ?

Update units in random order, one at a

time, to the state that gives the lowest

global energy 3 2 3 3

Stop when there is no unit that

changes its output 1 0 0

-1 -1

-E=goodness=3

Energy of a Hopfield Net

Start from a random state -4

1 0

Update units in random order, one at a

time, to the state that gives the lowest

global energy 3 2 3 3

Stop when there is no unit that

changes its output 1 0 0

-1 -1

-E=goodness=3

Energy of a Hopfield Net

Start from a random state -4

1 0

Update units in random order, one at a

time, to the state that gives the lowest

global energy 3 2 3 3

Stop when there is no unit that

changes its output ? 0 0

-1 -1

-E=goodness=3

Energy of a Hopfield Net

Start from a random state -4

1 0

Update units in random order, one at a

time, to the state that gives the lowest

global energy 3 2 3 3

Stop when there is no unit that

changes its output 1 0 0

-1 -1

-E=goodness=3

Energy of a Hopfield Net

Start from a random state -4

1 0

Update units in random order, one at a

time, to the state that gives the lowest

global energy 3 2 3 3

Stop when there is no unit that

changes its output 1 ? 0

-1 -1

-E=goodness=3

Energy of a Hopfield Net

Start from a random state -4

1 0

Update units in random order, one at a

time, to the state that gives the lowest

global energy 3 2 3 3

Stop when there is no unit that

changes its output 1 1 0

-1 -1

will not change its activation

Energy of a Hopfield Net

-4

The global minimum is given on the 0 1

right.

requires to change the activation of

two units at the same time

0 1 1

-1 -1

So, why not simultaneous change

activations of all units, depending on the

other? -E=goodness=5

Energy of a Hopfield Net

doing this could make the energy go up.

You can get oscillations

-100

0 0

5 5

1 1

I.e.: Simultaneous updating the top units will make the things worse

2. So sequential (asynchronous updating) is the one that gives the best results

Example taken from http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec11.pdf

Training a Hopfield Net

Training a Hopfield Net

energy minimum to a neural network with symmetric weights.

He proposed that the binary decision rule, can take partial memories

(when only some of the neurons get activated) and turn them into

complete memories.

Content addressable

Robust against hardware damage

Can be used for noise reduction, pattern recognition, pattern

completion

Training a Hopfield Net

In the previous slides, the weights were already set and we changed the

activation until we reached at a minimal energy

The question is: How do we set the weights, given an input, such that the minimal

energy corresponds to the network units having the same activations as the input

On the remaining of this course we will consider that units have activations of

(-1, 1) instead of (0,1)

Training a Hopfield Net

=

<

The network has the lowest energy (given a pattern ) when:

Units with same activation are linked with a positive weight

Units with different activation are linked with a negative weight

Training a Hopfield Net

+

? 1

If the weight is positive, then the contribution of unit to unit is given by the sign

of activation of unit : unit attracts unit to have same activation

If the weight is negative, then the contribution of unit to unit will have the

opposite sign of activation of unit : unit repels unit y

Training a Hopfield Net

Given these observations, the weights will be adjusted in the following way:

=

1 1

= ( )( )

2 2

Given more patterns, the rule for setting the weight is:

1

= =1

Training a Hopfield Net

Example:

and 2 = (1, 1, 1, 1, 1)

So we need to train a network with 5 inputs, where each node is connected with

each other, but not itself.

The weights will be represented as symetrical matrix with the elements on the

diagonal set to 0

0

0 0

0

Training a Hopfield Net

The first sample is presented to the network. 1 = (1,1,1, 1, 1)

-1 -1 1 1 -1 1

0 0 -1 0 0 0 0 0

1 1

0 1 0 0 0 0 0

0 0 1 0 0 0 0 0

0 -1 0 0 0 0 0

-1 1

1 0 0 0 0 0

Training a Hopfield Net

Each cell in the matrix, will be set to -1 or 1: the result of multiplying every two

activations.

Note: this is equivalent to computing

The elements on the main diagonal will be set to 0 (since there is no connection

from a unit to itself)

The result will be added (summation) to the global weight matrix

-1 1 1 -1 1

0 -1 -1 1 -1

-1 0 -1 -1 1 -1

-1 0 1 -1 1

1 -1 0 1 -1 1

-1 1 0 -1 1

1 -1 1 0 -1 1

1 -1 -1 0 -1

-1 1 -1 -1 0 -1

-1 1 1 -1 0

1 -1 1 1 -1 0

Training a Hopfield Net

The same thing happens with the second pattern: 2 = (1, 1, 1, 1, 1)

The values on the weight matrix will be in the interval [-M,M] where M represents

the number of patterns

1 -1 1 -1 1

0 -2 0 0 0

1 0 -1 1 -1 1

-2 0 0 0 0

-1 -1 0 -1 1 -1

0 0 0 -2 2

1 1 -1 0 -1 1

0 0 -2 0 -2

-1 -1 1 -1 0 -1

0 0 2 -2 0

1 1 -1 1 -1 0

Hopfield problem: Spurious

minima

Hopfield problem: Spurious minima

Spurious minima represents the situation where the network settles to a local

minimum that does not represents a learned pattern

configurations, merge together to create a new minima that has an even lower

energy

Hopfield problem: Spurious minima

Example. Given 3 patterns we can create a forth one. Each node will have a

value that is equal to the majority of the 3 patterns

1 = (1, 1, 1, 1, 1, 1, 1, 1, 1, 1) E=-45

2 = 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 E=-43

3 = (1, 1, 1, 1, 1, 1, 1, 1, 1, 1) E=-37

-----------------------------------------------------------------

4 = (1, 1, 1, 1, 1, 1, 1, 1, 1, 1) E=-43

Hopfield problem: Spurious minima

An interesting solution that was proposed by Hopfield et. all , was to use

unlearning:

Start from an initial random state

Let the network reach a minimum (which is probable to be a spurious

pattern)

Do the opposite of learning.

=

The authors showed that this worked, but had no analysis. For example, how

much unlearning should we do?

Other authors that have tested this idea found that the network ends up

working erratically (Cristos 1994)

Hopfield problem: Spurious minima

Francis Crick and Mitchison, proposed that human brains also do unlearning in

order to better store memories

They proposed that unlearning is what is going on in the brain during REM sleep

During the day you do lot of learning and you get spurious minima.

At night, you put the network to a random state (dreaming), you settle to a

local minima, and then you unlearn (forget you dream).

This solves a puzzle: why do we bother dreaming (the brain patterns looks

similar during the day as in REM sleep) if we forget what we dreamt.

Network Capacity

Network Capacity

The network capacity determines how many memories we can stored and

retrieved. Behind this limit spurious states are very probable.

Hopfield (and many other authors) have showed, that when Hebbian learning is

used, the capacity of the Network (using random patterns) is just

0.15N memories (0.138N more exactly)

Other authors (McEliece et al. (1987)) have shown that the capacity can not

exceed 4ln() , or 2 () is some errors are tolerable

Network Capacity

Amos Storkey (1997) modified the Hebbian learnig rule, and achieved a greater

capacity:

1 1 1 1

= +

where

1

=

=1,!=,

Network Capacity

Amos Storkey (1997)proved that his equation can boost the capacity of the

network to

2 ln( )

Method stores 5 times more

patterns

rule is that it has properties that

make it biologically plausible:

Is incremental

Is local

Network Capacity

The idea is to train the network such that every pattern is stable.

Initialize the neurons with a pattern

For each unit:

If it has the right value (given by the pattern) then leave it alone

If it doesnt, modify the weight coming to it using the perceptron rule

Do this for all the pattern in several iterations

Obviously, this loses the properties (it is no longer iterative) that would have make

it biologically realistic.

If there is a way to adjust weights (if training converges), than this method

provides the highest capacity

Questions & Discussion

References

https://www.doc.ic.ac.uk/~sd4215/hopfield.html

http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec11.pdf

https://en.wikipedia.org/wiki/Hopfield_network

http://homepages.inf.ed.ac.uk/amos/publications/Storkey1997IncreasingtheCapacityoftheH

opfieldNetworkwithoutSacrificingFunctionality.pdf

http://web.cs.ucla.edu/~rosen/161/notes/hopfield.html

http://neupy.com/2015/09/20/discrete_hopfield_network.html

http://www.comp.leeds.ac.uk/ai23/reading/Hopfield.pdf

http://perso.ens-lyon.fr/eric.thierry/Graphes2010/alice-julien-laferriere.pdf

Humayun Karim Sulehria, Ye Zhang, Study on the Capacity of Hopfield Neural Networks

Amos Storkey, Increasing the capacity of a Hopfield network without sacrificing functionality,

http://homepages.inf.ed.ac.uk/amos/publications/Storkey1997IncreasingtheCapacityoftheH

opfieldNetworkwithoutSacrificingFunctionality.pdf

- 22trans3Uploaded byPotnuru Vinay
- nervous system powerpointUploaded byabisantiago6131
- Outcome GmfmUploaded byCruzMilagrosQuispeLopez
- Prelim General PsychologyUploaded byEunice Domingo Alipio
- musicUploaded byapi-356316778
- Bug AlgorithmsUploaded byAsh
- Case Study on Pre EclampsiaUploaded byVianca Louriele Lavarro
- Transport ModelUploaded bybukenzo
- Germany.pdfUploaded byWilliam Gómez Ortega
- neuro assessmentUploaded byTran Ha
- Shortened Halstead Category TestUploaded byConstantinos Christodoulides
- Inteliigence is It in Brain or HeartUploaded byFatini Zulkifli
- StagesUploaded byhellowee
- r_magnus_1926.pdfUploaded bylaughAndCry72
- SCI SeminarUploaded byTeves Adrian
- BR 09_Siva_S P RobbinsUploaded bysivakumarsachin
- senior researchUploaded byapi-276180326
- Project PresentationUploaded byBetson Joseph George
- 1. Connected LearningUploaded bySandy Kontreraz
- Information and Communications Technology HandoutsUploaded byMyrna Monta
- project proposalUploaded byapi-453345074
- Algebraic Thinking Patterns PosterUploaded byvakileroaya
- Data for Impact AnalysisUploaded byapi-3816098
- AuditoryProcessing&Tinnitus 01Uploaded byWesley Jackson
- Scalp_EEGUploaded byEslam Elsayed
- Principles of CookingUploaded bySHERYL
- Skin-color Detection Based on Adaptive ThresholdsUploaded byKrunal Solanki
- PGDORM Sem 2 2017-18.pdfUploaded byZiroh
- Midterm QuestionsUploaded byBrooke Brewer
- Relative Resource ManagerUploaded bysamity

- Advanced Cluster AnalysisUploaded byWole Ajala
- Tune ParetoUploaded byMarius_2010
- Metric LearningUploaded byMarius_2010
- Chap10 Anomaly DetectionUploaded bymunich_1502
- Survery of nearest neighborUploaded byGanesh Chandrasekaran
- curs2.pdfUploaded byMarius_2010
- Build KD TreeUploaded byMarius_2010
- KD Skinny FatUploaded byMarius_2010
- Curs10 SiteUploaded byMarius_2010
- A Comparison between Memetic algorithm and Genetic algorithm for the cryptanalysis of Simplified Data Encryption Standard AlgorithmUploaded byAIRCC - IJNSA
- Rice Price PredictionUploaded byMarius_2010
- Non-param Classification BallTreeUploaded byMarius_2010
- PricesUploaded byMarius_2010
- xgboost-150831021017-lva1-app6891Uploaded byPushp Toshniwal
- BoostedTree.pdfUploaded bysleepyhollowinoz
- curs1siteUploaded byMarius_2010
- WNvanWieringen HDDA Lecture4 RidgeRegression 20162017Uploaded byMarius_2010
- r Mals ChainsUploaded byMarius_2010
- IntroMAs-advanced.pdfUploaded byMarius_2010
- curs7.pdfUploaded byMarius_2010
- MurrellPaths.pdfUploaded byMarius_2010
- PE_Tutorial.pdfUploaded byMarius_2010
- 4. hierarchical clustering.pdfUploaded byMarius_2010
- dtree18.pdfUploaded byMarius_2010
- seq-patUploaded byMarius_2010
- Chap1 IntroUploaded byTSMFORU

- nnt_introUploaded byLeonid Alemán Gonzales
- Applications of Linear AlgebraUploaded bySolitary ChOcoholic
- Numerical OptimizationUploaded byMS
- NUMERICAL ANALYSIS-4. ROOT-FINDING (OVERVIEW).pdfUploaded byJohn Joseph Resngit Raymundo
- MATH_F313_1471Uploaded byKoustubh Gupta
- A Brief Description of the Levenberg-Marquardt Algorithm Implemened by LevmarUploaded byk3ndr0
- 0618783768_138794Uploaded byJackson Lam
- Lecture 2aUploaded bySam Keays
- Neuro fuzzyUploaded byAdam Anugrah
- 44852485 Numerical MethodsUploaded byDISCOVERY
- NN for Pattern for ClassificationUploaded byVivek Goyal
- IEA-EORUploaded byYinzhang
- CFD Question Bank for AUC R2008_SUploaded byGurunath Aero
- NUmerical Method Tutorial Final -1Uploaded bysushant giri
- Chapter6(MBI)ElUploaded byjokowi123
- MIT Introduction to Deep Learning Course Lecture 1Uploaded bytintojames
- 1512.08301v1Uploaded bywangxy802
- LESSON PLAN_MCS 301.docxUploaded byjoydeep12
- A introduction to svm.pdfUploaded byHau Nguyen
- Civil Engineer IV Sem Set 3 (1)Uploaded byUma Subramanian
- Two Layers Soil Stratification Using NumericalUploaded byRaul Marihuan González
- CONM TechMax SyllabusUploaded byshyjuother9773
- Project ReportUploaded bysaksham
- Pages From Zhang2015Uploaded byKAS Mohamed
- pattern classification labreportUploaded bySabitParvez
- the remainder and factor theoremsUploaded byapi-285179261
- Numerical MethodsUploaded bymojiryhamid
- Assignment 1Uploaded byjohnyrock05
- Image Reconstruction Using Multi Layer Perceptron Mlp and Support Vector Machine Svm Classifier and Study of Classification AccuracyUploaded byIJSTR Research Publication
- Implementasi Jaringan Saraf Tiruan Dengan MATLABUploaded byUsep Siregar