0 views

Uploaded by Tariq Zuhluf

e 033028

- Notes on Backpropagation
- Neural Network EOR
- Lec5.1_Convolutional-networks-for-images_-speech_-and-time-series.pdf
- UT Dallas Syllabus for cs4375.501.07f taught by Yu Chung Ng (ycn041000)
- Modular Neural Network Approach for Short Term Flood Forecasting a Comparative Study
- Smart Number Plate Identification Using Back Propagation Neural Network
- Automatic grain inspection
- EJSR_78_1_09
- Paper
- ajeba2127-34
- Neural Control
- ES98BW
- Introduction to Artificial Neural Network (ANN)
- Evolution of Neural Net Architectures
- Fuzzy C Means
- VAriable Speed Turbines
- A Simplified Design of Multiplier for Multi Layer Feed Forward Hardware Neural Networks
- Performance Evaluation using Supervised Learning Algorithms for Breast Cancer Diagnosis
- Appendices 2 ADDITIONAL CASES
- J&M-18

You are on page 1of 5

Networks (3) ur

co www. http://

w

eb

compi rg

nt/pag u.ac.uk/eng

sit

e.cfm? /

pge=6

e

732

By Chris MacLeod and Grant Maxwell

This month we look at two more advanced neural nets. The Hopfield

network, which uses feedback in its structure, and the Competitive net

which can recognise patterns in data, even if the programmer doesn’t

know they exist.

Forward pass operation fed back to the input once it’s been

Feedback paths calculated and so goes through the

The neurons operate in the same network again. Eventually, if the

way as the binary ones described in network has been trained properly,

Input Output 1

part 1 (except that they produce a the output will become constant

A

–1 and +1 output, rather than 0 and (the inputs and outputs will be the

1). The only difference in operation same). The network is said to have

is that the output of the network is relaxed. The network is then ready

Input

B Output 2

Start – Apply

020324 - 3 - 11 initial inputs to

network

Figure 1. A Hopfield Net.

published a famous paper on neural nets.

This paper helped to re-ignite the field,

which had been languishing in the doldrums

for some time.

Are Outputs the Yes Stop and read

same as inputs? final outputs

Actually, the ANN which bears his name

— the Hopfield Network — is of somewhat

limited practical use, but it does help us to

understand the ins and outs of neural net No

behaviour.

What Hopfield did was to create a net- Make current Next Inputs to network =

work with feedback — with connections from Current Outputs

the outputs, back towards the inputs. Figure

020324 - 3 - 12

1 shows the idea. The network always has a

single layer and the same number of inputs

as neurons. Figure 2. Running data through a Hopfield net.

COMPUTING

shows the idea.

Once the network is trained properly, all

we have to do is present it with the corrupted

version as its inputs and then wait until the

network stops cycling as described above.

Input to network Output after network Once this has happened, we can read the

has relaxed outputs of the network and they will give us a

reconstructed image, see Figure 4.

020324 - 3 - 13

In the original Hopfield net, all inputs and

outputs are –1, which could represent, say, a

Figure 3. Operation of a Hopfield net. white pixel and +1 for a black pixel. Net-

works with continuous outputs are today

more common, but for our discussion, we’ll

stick with the simple case.

Input Output 1

A Training

Now that we know what the Hopfield net-

Input

work does, let us turn our attention to how it

B Output 2 can be trained.

Apply input Retrieve re-

Compared with the Back Propagation net-

(corrupted) constructed

image here. image here. work, training the Hopfield is easy. All the

weights are calculated using a simple for-

020324 - 3 - 14 mula:

weights Wn,n = 0.

for you to read its outputs. Figure 2 the Hopfield network can do, which Where Wm,n is the weight of the connection

shows this process in the form of a the BP network can not. A Hopfield between the mth input and the nth neuron

flow chart. network, rather than just recognis- and On is the nth output desired from the net-

ing an image, can store and retrieve work.

patterns — it has a memory. We can In other words, to find the weight of the

Uses input a corrupted image into the connection between input m and neuron n,

Before going any further, it’s worth network and it will reproduce the take each pattern to be trained in turn and

pausing to consider what it is that perfect stored version. Figure 3 multiply the mth output by the nth output and

add them all together. As usual this is best

illustrated by example, see Figure 5.

Let’s say we’d like to train three patterns:

1

w1,1 OA

w2,1

w1,3

w3,1

Pattern number two:

w2,2 2 OB

Pattern number three:

w2,3 w3,2

w1,1 = 0

3 OC w1,2 = OA(1) × OB(1) + OA(2) × OB(2) + OA(3) ×

w3,3 OB(3) = (–1) × (–1) + 1 × (–1) + –-1) × 1 =

020324 - 3 - 15 –1

w1,3 = OA(1) × OC(1) + OA(2) × OC(2) + OA(3) ×

OC(3) = (–1) × 1 + 1 × (–1) + (–1) × 1 = –3

Figure 5. Worked example of Hopfield training.

COMPUTING

w2,2 = 0

w2,1 = Ob(1) × Oa(1) + Ob(2) × Oa(2) + Ob(3) ×

O

Oa(3) = (–1) × (–1) + (–1) × 1 + 1 × (–1) = I U

–1 N T

P

w2,3 = Ob(1) × Oc(1) + Ob(2) × Oc(2) + Ob(3) × U

P

Oc(3) = (–1) × 1 + –1) × (–1) + 1 × 1 = 1 T

U

T

S S

w3,3 = 0

w3,1 = Oc(1) × Oa(1) + Oc(2) × Oa(2) + Oc(3) ×

Oa(3) = 1 × (–1) + (–1) × 1 + 1 × (–1) = –3 020324 - 3 - 16

Ob(3) = 1 × (–1) + (–1) × (–1) + 1 × 1 = 1 Figure 6. A general neural net.

only once and are not repeated.

We can write a simple algorithm to set the Listing 1

weights for a Hopfield as shown in Listing 1. FOR f = 1 TO no_of_inputs

Where the same variables are used as FOR t = no_of_inputs + 1 TO no_of_inputs + no_of_outputs

shown in the forward pass example in part 1 FOR p = 1 TO no_of_patterns

(where the weights are held in a two-dimen- w(f, t) = w(f, t) + i(p, f) * i(p, t - no_of_inputs)

sional array). The desired outputs are held in NEXT p

an array i(pattern_no, pixel_number). IF t = no_of_inputs + f THEN w(f, t) = 0

NEXT t

Capabilities NEXT f

else can it do? Actually, its practical applica-

tions are a little limited, but it tells us a lot

about the general capabilities of neural nets. Input 1 Input 2

In part 1, we discussed the similarity of

the feed forward network to combinational

logic. But the ANN is logic which can produce

any truth table by learning, rather than

detailed design. Similarly, the analogy for the

Hopfield is sequential logic. After all, a

flip/flop like a JK or SR is a simple memory 1

and this is also achieved through the use of

2 3

feedback.

In fact, the Hopfield can produce time-

series, oscillatory or even chaotic outputs if

Output 1 Output 2 Output 3

you let it; although the training illustrated

above is designed always to produce a stable 020324 - 3 - 17

steady state. Figure 7. A simple competitive net.

The simple Hopfield net illustrated here

has limitations. It is prone to local minima

(which in this case means that it may have Input 1 Input 2

difficulty reconstructing some patterns), so

more sophisticated training algorithms have

been developed. For example, there are vari-

ants of the Back Propagation algorithm which

can be used to train Hopfield nets using tar-

gets (like the BP networks in part 2), but

these don’t guarantee stability.

We can extend the capabilities of the sim- 1 2 3

ple Hopfield if we add an extra layer. Such

networks are known as Bi-directional Asso-

ciative Memories (BAMs) and can associate

an input with a different memory. But beyond Output 1 Output 2 Output 3

this, the structure of the Hopfield net is too 020324 - 3 - 18

rigid, we need to use its lessons to devise

more general nets. Figure 8. A winning neuron.

COMPUTING

General Neural Nets We won’t worry too much at this

stage about the set up of the

We’ve seen how the Hopfield net is weights except to say that they are

more general than the simple feed-

forward type. In fact the feedback

type just degenerates to a feedfor-

essentially random.

Now let us apply a pattern to the

network. Just by chance (since the

Input 1

{ Vector of length L

{

ward net if the feedback paths are weights are random), one of these

set to zero. neurons will have a higher output Input 2

You might guess therefore, that than the others — let’s say it’s neu-

the most general neural nets would ron three, as shown in Figure 8.

have a mixture of both feedback and We say that this neuron has won

020324 - 3 - 19

feedforward connections. In fact this and set its output to 1 and the oth-

is true. In the most general network, ers to zero.

any neuron could be connected to Now we train only the weights of Figure 9. The inputs shown on a graph.

any other, Figure 6 shows the idea. neuron 3 (the ones shown in bold),

Training such networks is tricky, so that, if this pattern comes along

as algorithms like the Hopfield Train- again it will have an even higher out-

ing illustrated above and even Back put — it will win even more easily.

Propagation only operate when the So neuron three will always fire Weight vector length W

network has a defined and limited

structure. To train a network where

any neuron may be connected to any

when this pattern comes along. In

other words, neuron three recognises

the pattern. This is very simple to

Weight 1 { Input vector length L

{

other demands more advanced algo- do; we just update the weights with Weight 2

rithms. this formula:

Perhaps the easiest algorithm to

employ and certainly the most com- W+ = W + η(Input – W)

mon in such circumstances is the 020324 - 3 - 20

Genetic Algorithm. One can employ Where W+ is the new (trained

the algorithm to choose the weights weight) and W is the original

in a general network in the same weight, Input is the input feeding Figure 10. The weight vector of neuron 3.

way as one can use it to choose com- that weight and η is a small con-

ponent values in the examples given stant, much less than 1 (say 0.1).

in that article, the fitness of the net- Of course if another, completely (even although the similarity may not be obvi-

work being the inverse of its error. different, pattern comes along a dif- ous to the user). We can say therefore that,

The details of such advanced train- ferent neuron will win and then this whereas the Back Propagation network is

ing methods can wait for a future new neuron will get trained for that trained by the user to recognise patterns, the

article. pattern and so the network self Competitive net trains itself to classify pat-

organises so that each neuron fires terns.

for its own pattern. Of course you could use the competitive

Competitive Learning neuron to recognise patterns like Back Prop-

Now, let’s look at a quite different agation. But this seems rather a waste of

network. You’ll remember that in Uses effort since BP works extremely well and is

part 2, we mentioned that probably Suppose we let a competitive net- generally easier to set up than a competitive

80% of neural nets used in practice work loose on some data — let’s say net.

were Feedforward, Back Propagation from the financial markets. The net-

nets. This leaves the question of the work would organise itself to find

remaining 20%. Well, most of these patterns in the data. Exactly what More detail

are of a type of network known as a these patterns are, we don’t know, To understand some of the subtle features of

Competitive or Winner-Takes-All the network decides for itself — we the competitive system, we need to examine

Net. don’t give it examples like a Back its operation a little more closely. To do this,

Propagation network. This is both let’s look at the network shown in Figures 7

the attraction and the disadvantage and 8 more closely.

Operation of the Competitive network — you The network has two inputs and it’s pos-

The Competitive net is best illus- might find an important pattern in sible to represent these as a line (called a vec-

trated by example. Suppose we have the data which you didn’t know tor) on a graph, where y is the value of input

a network of three neurons as shown existed — but it could miss what 1 and x input 2. This is shown in Figure 9 (of

in Figure 7. you’re looking for and find some course this applies to any number of inputs,

The neurons work in exactly the other unwanted patterns to recog- but two are easy to visualise).

same way as those already nise. The length of this vector by Pythagoras is:

described in part 1, except that we In the same way and related to

don’t need to apply a threshold or this, the network will fire the same Length = ( input1) 2 + ( input 2) 2

sigmoid function. neuron for patterns it finds similar

COMPUTING

We can also plot a line representing the

weights of neuron 3 on the same graph by

Weight3 vector

making its two weights the x and y coordi- Weight1 vector

nates, as shown in Figure 10. Input vector length L

Now, when we work out the output of the

neuron (i1w1 + i2w2), what we are actually

doing is calculating what’s known as the dot

Weight2 vector All vector lengths are one

product — which can be considered a mea- unit (so they all lie on this

sure of how similar the two vectors are. If circle).

both the vectors were exactly the same (one

020324 - 3 - 21

lying on top of the other) the output would be

larger than if they were different.

If all the vectors were the same length, Figure 11. Weight vectors for all three neurons.

then we’d just be measuring the angle

between them (which would make matters

easier, as it means that we don’t have to take Training moves Weight3

length into consideration), so that is what we vector towards input vector

do. We can make all the vectors one unit long

by dividing them by their length. Weight1 vector

Now, consider the weight vectors for all Input vector length L

three neurons in the network, Figure 11.

These have all been normalised to one unit as

described above.

Neuron 3 has won because it is closest to Weight2 vector

the input and therefore has the largest dot

020324 - 3 - 22

product (it is most similar to the input). What

the training does, is move the weight vector

of neuron 3 even closer to the input, as

shown in Figure 12 (remember that only the Figure 12. Effect of training.

weights of neuron 3 are trained).

This, of course, makes it likely that, if a

similar pattern comes around again, neuron

3 will fire. “winning” neuron,

The training formula W+ = W + η(Input – fully trained.

W) doesn’t preserve the unit length of the

weight vector it’s operated upon; so after

using it, you should divide the weight vector

of the winning neuron by its length to make Adjacent neurons

it one unit again. partially trained.

You can probably see that the distribution

of the weights in this type of network is quite

critical and so it helps to consider the distri- 020324 - 3 - 23

bution of vectors around the origin when set-

ting up the network to ensure an even cover-

age. Figure 13. A Self Organising Map.

on Competitive Neurons are grouped together and are far

Competitive neurons are seldom used just on away from the less similar ones.

there own, but form the mainstay of several Another very advanced network

more complex networks. They are often laid based on Competitive neurons is

out in the form of a 2D grid as shown in Figure Adaptive Resonance Theory (ART).

13. This is known as a Kohonen Self Organ- This network can change its size

ising Map. and grow as it learns more patterns.

What happens in this case is that the win- In the final part of the series we’ll

ning neuron (shown in black) is fully trained have a look at some of the other

and the surrounding neurons (shown in grey) applications of neural nets and some

are partially trained (by making η in the for- of the more advanced topics which

mula, a smaller number). researchers are wrestling with.

When the network has been allowed to (020324-3)

train in this way the result is that it forms a

- Notes on BackpropagationUploaded byJun Wang
- Neural Network EORUploaded byAmr Hegazy
- Lec5.1_Convolutional-networks-for-images_-speech_-and-time-series.pdfUploaded byJithesh Gopal
- UT Dallas Syllabus for cs4375.501.07f taught by Yu Chung Ng (ycn041000)Uploaded byUT Dallas Provost's Technology Group
- Modular Neural Network Approach for Short Term Flood Forecasting a Comparative StudyUploaded byEditor IJACSA
- Smart Number Plate Identification Using Back Propagation Neural NetworkUploaded byAnonymous vQrJlEN
- Automatic grain inspectionUploaded bycuterohit16
- EJSR_78_1_09Uploaded byRaul Mamani
- PaperUploaded byPushparaj Karu
- ajeba2127-34Uploaded byMiftakhul Falah
- Neural ControlUploaded byArioston Júnior
- ES98BWUploaded byMustaf Mohamed
- Introduction to Artificial Neural Network (ANN)Uploaded byAnita Wulansari
- Evolution of Neural Net ArchitecturesUploaded byDavid Henriques
- Fuzzy C MeansUploaded byYudhie Prabowo
- VAriable Speed TurbinesUploaded bymuazeemK
- A Simplified Design of Multiplier for Multi Layer Feed Forward Hardware Neural NetworksUploaded byInternational Journal of Research in Engineering and Technology
- Performance Evaluation using Supervised Learning Algorithms for Breast Cancer DiagnosisUploaded byIRJET Journal
- Appendices 2 ADDITIONAL CASESUploaded byraum123
- J&M-18Uploaded by1wocker1
- OJCST_Vol10_N2_p_391-399Uploaded byRaju KJ
- Call Other ProgramsUploaded byjfranbripi793335
- Class_13_8.pdfUploaded byRam Bhagat Soni
- Enhancement of power quality by DVR using "ANN Technique" under unbalanced and non-linear loadsUploaded byInternational Journal of computational Engineering research (IJCER)
- [Jagannathan_Sarangapani]_Neural_network_control_o(b-ok.org).pdfUploaded byJosmir S. Rodríguez K.
- 10. Comp Sci - IJCSEITR - A New Approach for Person Verification - A D Bhople_New1Uploaded byTJPRC Publications
- Steady State Modelling of Chemical Systems by Using Neural NetworksUploaded byAmit Vaghasia
- PU Chronicles BITS PilaniUploaded byVishal Mittal
- AP 4101238242Uploaded bydarshak444
- Fábio Ghignatti 2002Uploaded byNafri Irfan

- e 006050Uploaded byTariq Zuhluf
- e 919046Uploaded byTariq Zuhluf
- e 905030Uploaded byTariq Zuhluf
- e 047041Uploaded byTariq Zuhluf
- e02ch54Uploaded byTariq Zuhluf
- e 986024Uploaded byTariq Zuhluf
- e 055077Uploaded byTariq Zuhluf
- e01c093Uploaded byTariq Zuhluf
- e01c093Uploaded byTariq Zuhluf
- e91c061Uploaded byTariq Zuhluf
- e94c016Uploaded byTariq Zuhluf
- e92c008Uploaded byTariq Zuhluf
- e 037028Uploaded byTariq Zuhluf
- e05b040Uploaded byTariq Zuhluf
- e05b040.pdfUploaded byTariq Zuhluf
- e006050.pdfUploaded byTariq Zuhluf
- e 907056Uploaded byTariq Zuhluf
- e90cS25Uploaded byTariq Zuhluf
- e02ch04Uploaded byTariq Zuhluf
- e02ch43Uploaded byTariq Zuhluf
- e01ch25Uploaded byTariq Zuhluf
- e01c098Uploaded byTariq Zuhluf
- e01ch09Uploaded byTariq Zuhluf
- e03b035Uploaded byTariq Zuhluf
- e051042.pdfUploaded byTariq Zuhluf
- e94c100Uploaded byTariq Zuhluf
- e92b021Uploaded byTariq Zuhluf
- e05c070.pdfUploaded byTariq Zuhluf
- e92c046Uploaded byTariq Zuhluf

- Project #7 - HypertensionUploaded byKP
- As if Jesus - Adrian a BantjesUploaded byJuan Antonio Fernandez Velazquez
- dreikurs.docxUploaded byAnonymous TVqM5dez
- Chest Tube and WSDUploaded byAnita Zhang
- SIE-07-008_Lantis_v6Uploaded byMerhan Fouda
- How to PrepareUploaded byPrc Santiago-City
- Systems Development Cycle - Software Project ManagementUploaded bysavassakar
- Teachers GuideUploaded byFrancisco Ruiz
- Town of Lakeshore relocates their legal office 9 times in 7 years.Uploaded byLakeshore's legal counsel: Walstedt Renick LLP
- Communication SkillUploaded byedudivya
- IPCRF-2018.final (2)Uploaded byMarietta Argao
- Gam if IcationUploaded byElma Bautista
- Reference USUUploaded byTania AR
- Ilocos NorteUploaded byFreya Rafanan
- DMUploaded byjhoan
- Classroom management.pdfUploaded byDan GROSU
- Vision of an Islamic SchoolUploaded byMaulviJahangirMahmud
- Performance+Analysis+of+Spectrum+Sensing+Techniques+for+Cognitive+Radio+over+Wireless+Fading+ChannelsUploaded byHendry Hardiyanto
- Definition of ManUploaded bydeocl
- A Study on Cash Conversion Cycle With SpecialUploaded byRajith Radhakrishnan
- Greeting and General ThingsUploaded bySuhanto Kastaredja
- Vocabulary for Generalizing a StatementUploaded byDarun Run
- Independent Writing Template and Sample EssaysUploaded byAnonymous XqZ2YI
- Perio 2Uploaded byAbdallah Essam Al-Zireeni
- Digital Booklet - CalifornicationUploaded byChristian Campos
- b.a papersUploaded byMuhammad Aslam
- Building Brand Equity and Brand IdentityUploaded byMetta Yunita
- A Candidate of New Insensitive High Explosive- MTNIUploaded byPhileas
- Effective Response to the Arab Spring: Recommendations for American Foreign PolicyUploaded bySaed Kakei
- Nursing Care Management 106Uploaded bymaynard baird