82 views

Uploaded by Kshirod Kumar Dash

- Artificial Neural Networks (ANN)
- Neural Networks
- Neural Networks
- Neural Networks
- Chapter 6 - Feedforward Deep Networks
- Neural_Computing___An_Introduction.pdf
- NeuralNetworkFAQ
- Multilayered Feed Forward Artificial Neural Network Model
- Investigation of Phytoplankton Dynamics Using Time Series Analysis of Biophysical Parameters in Ossie
- medical image classification
- Proceedings_NSAI
- Forecasting the Tehran Stock Market by Artificial Neural Network
- Sonar-Based Mobile Robot Navigation Through Supervised Learning on a Neural Net
- Still an Alien Planet in Soft Computing
- DL501 Course Summary
- Chapter 5 ANN
- The Nature of Code - Chapter 10
- A Basic Introduction To Neural Networks.doc
- PFC Dynamic Neural Networks for MBS in Mechanical Systems.pdf
- Geodezija_2012_04_Chapter6

You are on page 1of 45

Vincent Cheung

Kevin Cannons

Signal & Data Compression Laboratory

Electrical & Computer Engineering

University of Manitoba

Winnipeg, Manitoba, Canada

Advisor: Dr. W. Kinsner

Neural Networks

Outline

● Fundamentals

● Classes

● Design and Verification

● Results and Discussion

● Conclusion

Cheung/Cannons 1

Fundamentals Neural Networks

Classes

► Transforms inputs into outputs to the best of its ability

Design

Inputs Outputs

Results

Inputs NN Outputs

Cheung/Cannons 2

Fundamentals Neural Networks

Classes

Design

Results

Cheung/Cannons 3

Fundamentals Neural Networks

● Classification

Classes

matching

● Noise Reduction

► Recognize patterns in the inputs and produce

Design

noiseless outputs

● Prediction

► Extrapolation based on historical data

Results

Cheung/Cannons 4

Fundamentals Neural Networks

● Ability to learn

Classes

► Determine their function based only upon sample inputs

● Ability to generalize

i.e. produce reasonable outputs for inputs it has not been

Design

►

taught how to deal with

Results

Cheung/Cannons 5

Fundamentals Neural Networks

Classes

i1 w1

w2

i2

w3 Neuron Output = f(i1w1 + i2w2 + i3w3 + bias)

Design

i3

bias

Results

the computation of the outputs of all the neurons

► An entirely deterministic calculation

Cheung/Cannons 6

Fundamentals Neural Networks

Activation Functions

Classes

● Majority of NN’s use sigmoid functions

► Smooth, continuous, and monotonically increasing

(derivative is always positive)

Design

■ Consider “ON” to be slightly less than the max and “OFF” to

be slightly greater than the min

Results

Cheung/Cannons 7

Fundamentals Neural Networks

Activation Functions

Classes

logistic function

► f(x) = 1/(1 + e-x)

► The calculation of derivatives are important for neural

networks and the logistic function has a very nice

Design

derivative

■ f’(x) = f(x)(1 - f(x))

hyperbolic tangent

Results

►

► arctangent

the abilities of the neural network

Cheung/Cannons 8

Fundamentals Neural Networks

Classes

● Training is the act of presenting the network with

some sample data and modifying the weights to

better approximate the desired function

Design

► Supervised Training

■ Supplies the neural network with inputs and the desired

Results

outputs

■ Response of the network to the inputs is measured

The weights are modified to reduce the difference between

the actual and desired outputs

Cheung/Cannons 9

Fundamentals Neural Networks

► Unsupervised Training

Classes

■ The neural network adjusts its own weights so that similar

inputs cause similar outputs

The network identifies the patterns and differences in the

inputs without any external assistance

Design

● Epoch

■ One iteration through the process of providing the network

with an input and updating the network's weights

■ Typically many epochs are required to train the neural

Results

network

Cheung/Cannons 10

Fundamentals Neural Networks

Perceptrons

Classes

● Input neurons typically have two states: ON and OFF

● Output neurons use a simple threshold activation function

Design

► Limited applications

.5

Results

.2

.8

Cheung/Cannons 11

Fundamentals Neural Networks

Classes

adjusted according to the formula:

■ wnew = wold + α(desired – output)*input α is the learning rate

Design

1 0.5

0.2

0 1

1 0.8

Results

Æ update the weights

1 * 0.5 + 0 * 0.2 + 1 * 0.8 = 1.3

Assuming Output Threshold = 1.2 Assume α = 1

1.3 > 1.2 W 1new = 0.5 + 1*(0-1)*1 = -0.5

W 2new = 0.2 + 1*(0-1)*0 = 0.2

W 3new = 0.8 + 1*(0-1)*1 = -0.2

Cheung/Cannons 12

Fundamentals Neural Networks

Classes

► Multiple layers

■ The addition of one or more “hidden” layers in between the

input and output layers

Design

■ Usually a sigmoid function

► A general function approximator

■ Not limited to linear problems

Results

► The outputs of one layer act as inputs to the next layer

Cheung/Cannons 13

Fundamentals Neural Networks

XOR Example

Classes

Inputs

Output

Design

Inputs: 0, 1

Results

H1: Net = 0(4.83) + 1(-4.83) – 2.82 = -7.65 H2: Net = 0(-4.63) + 1(4.6) – 2.74 = 1.86

Output = 1 / (1 + e7.65) = 4.758 x 10-4 Output = 1 / (1 + e-1.86) = 0.8652

Output = 1 / (1 + e-2.187) = 0.8991 ≡ “1”

Cheung/Cannons 14

Fundamentals Neural Networks

Backpropagation

Classes

● A form of supervised training

● The basic backpropagation algorithm is based on

Design

derivatives of the error function

► Simple

► Slow

Results

Cheung/Cannons 15

Fundamentals Neural Networks

Backpropagation

Classes

square error:

E = (target – output)2

Design

► Output Neurons:

let: δj = f’(netj) (targetj – outputj) j = output neuron

∂E/∂wji = -outputi δj i = neuron in last hidden

Results

► Hidden Neurons:

j = hidden neuron

let: δj = f’(netj) Σ(δkwkj)

i = neuron in previous layer

∂E/∂wji = -outputi δj k = neuron in next layer

Cheung/Cannons 16

Fundamentals Neural Networks

Backpropagation

Classes

backpropagation

● These derivatives point in the direction of the

maximum increase of the error function

Design

direction will result in the maximum decrease of

the (local) error function:

Results

where α is the learning rate

Cheung/Cannons 17

Fundamentals Neural Networks

Backpropagation

Classes

► Too small

■ Convergence extremely slow

► Too large

■ May not converge

Design

● Momentum

► Tends to aid convergence

► Applies smoothed averaging to the change in weights:

Results

► Acts as a low-pass filter by reducing rapid fluctuations

Cheung/Cannons 18

Fundamentals Neural Networks

Local Minima

Classes

error function

► Key problem is avoiding local minima

► Traditional techniques for avoiding local minima:

■ Simulated annealing

Design

■ Genetic algorithms

Use the weights as chromosomes

Apply natural selection, mating, and mutations to these

chromosomes

Results

Cheung/Cannons 19

Fundamentals Neural Networks

Classes

● Not as general as backpropagation

● Made up of three layers:

Design

► Input

► Kohonen

► Grossberg (Output)

Results

Layer Layer Layer

Cheung/Cannons 20

Fundamentals Neural Networks

● Kohonen Layer:

Classes

inputs received

► The neuron with the largest sum outputs a 1 and the

other neurons output 0

Design

● Grossberg Layer:

► Each Grossberg neuron merely outputs the weight of the

connection between itself and the one active Kohonen

neuron

Results

Cheung/Cannons 21

Fundamentals Neural Networks

Classes

networks

● Each layer has its own distinct purpose:

► Kohonen layer separates inputs into separate classes

■ Inputs in the same class will turn on the same Kohonen

Design

neuron

► Grossberg layer adjusts weights to obtain acceptable

outputs for each class

Results

Cheung/Cannons 22

Fundamentals Neural Networks

Training a CP Network

Classes

► Input vectors are often normalized

► The one active Kohonen neuron updates its weights

according to the formula:

Design

where α is the learning rate

Results

closely match the values of the inputs

■ At the end of training, the weights will approximate the

average value of the inputs in that class

Cheung/Cannons 23

Fundamentals Neural Networks

Training a CP Network

Classes

► Weight update algorithm is similar to that used in

backpropagation

Design

Results

Cheung/Cannons 24

Fundamentals Neural Networks

Classes

discontinuous

● The number of neurons is very important:

Design

► Too few

■ Underfit the data – NN can’t learn the details

► Too many

■ Overfit the data – NN learns the insignificant details

Results

results are obtained

Cheung/Cannons 25

Fundamentals Neural Networks

Overfitting

Classes

Design

Training

Test

W ell fit

Overfit

Results

Cheung/Cannons 26

Fundamentals Neural Networks

Classes

not chosen

● What constitutes a “good” training set?

► Samples must represent the general population

Samples must contain members of each class

Design

►

► Samples in each class must contain a wide range of

variations or noise effect

Results

Cheung/Cannons 27

Fundamentals Neural Networks

Classes

► Eg. 10 inputs, 5 hidden neurons, 2 outputs:

► 11(5) + 6(2) = 67 weights (variables)

► If only 10 training samples are used to determine these

Design

■ Any solution found will be specific to the 10 training

samples

■ Analogous to having 10 equations, 67 unknowns Æ you

Results

can come up with a specific solution, but you can’t find the

general solution with the given information

Cheung/Cannons 28

Fundamentals Neural Networks

Classes

► Training set

■ A group of samples used to train the neural network

► Testing set

Design

neural network

■ Used to estimate the error rate

Results

Known Samples

Training Testing

Set Set

Cheung/Cannons 29

Fundamentals Neural Networks

Verification

Classes

network

● Common error is to “test” the neural network using

the same samples that were used to train the

neural network

Design

obviously perform well on them

► Doesn’t give any indication as to how well the network

will be able to classify inputs that weren’t in the training

Results

set

Cheung/Cannons 30

Fundamentals Neural Networks

Verification

Classes

results of the testing set

► Mean square error, SNR, etc.

Design

► Basic idea is to iterate the training and testing

procedures multiple times

Results

■ Cross-Validation

■ Bootstrapping

Cheung/Cannons 31

Fundamentals Neural Networks

operation of a perceptron

Classes

information about a face – the individual’s hair,

eye, nose, mouth, and ear type

Design

■ +1 indicates a “girl” feature

■ -1 indicates a “guy” feature

Results

and a boy otherwise

Cheung/Cannons 32

Fundamentals Neural Networks

Classes

output

Design

Feature neurons neuron indicating

Input boy or girl

Values

Results

possible inputs over 1000 epochs

● The perceptron was able to classify the faces that

were not in the training set

Cheung/Cannons 33

Fundamentals Neural Networks

multilayer feedforward NN’s with a single hidden

Classes

► Inverter

■ The NN was trained to simply output 0.1 when given a “1”

and 0.9 when given a “0”

Design

■ 1 input, 1 hidden neuron, 1 output

■ With learning rate of 0.5 and no momentum, it took about

3,500 epochs for sufficient training

■ Including a momentum coefficient of 0.9 reduced the

Results

Cheung/Cannons 34

Fundamentals Neural Networks

► Inverter (continued)

Classes

without hampering convergence for this simple example

■ Increasing the epoch size, the number of samples per

epoch, decreased the number of epochs required and

seemed to aid in convergence (reduced fluctuations)

Design

number of epochs required

Allowed the NN to better memorize the training set – the goal

of this toy problem

Not recommended to use in “real” problems, since the NN

Results

Cheung/Cannons 35

Fundamentals Neural Networks

► AND gate

Classes

■ About 2,500 epochs were required when using momentum

► XOR gate

■ Same as AND gate

Design

► 3-to-8 decoder

■ 3 inputs, 3 hidden neurons, 8 outputs

■ About 5,000 epochs were required when using momentum

Results

Cheung/Cannons 36

Fundamentals Neural Networks

Classes

function, |sin(x)|, and to generalize

■ 1 input, 5 hidden neurons, 1 output

■ The NN was trained with samples between –π/2 and π/2

The inputs were rounded to one decimal place

Design

■ The test data contained samples in between the training

samples (i.e. more than 1 decimal place)

The outputs were translated back to between 0 and 1

Results

■ Not smooth function at 0 (only piece-wise continuous)

Cheung/Cannons 37

Fundamentals Neural Networks

2

► Gaussian function approximator (e-x )

Classes

■ Similar to the absolute sine function approximator, except

that the domain was changed to between -3 and 3

■ About 10,000 epochs were required with momentum

■ Smooth function

Design

Results

Cheung/Cannons 38

Fundamentals Neural Networks

► Primality tester

Classes

■ The input to the NN was a binary number

■ The NN was trained to output 0.9 if the number was prime

and 0.1 if the number was composite

Classification and memorization test

Design

■ About 50,000 epochs required for the NN to memorize the

classifications for the training set

No attempts at generalization were made due to the

Results

■ Some issues with local minima

Cheung/Cannons 39

Fundamentals Neural Networks

■ Provide the network with a seed, and a prime number of the

Classes

■ 7 inputs, 4 hidden neurons, 7 outputs

■ Both the input and outputs were binary numbers

■ The network was trained as an autoassociative network

Design

and it was requested that the network echo the prime

numbers

The intent was to have the network output the closest prime

number when given a composite number

■ After one million epochs, the network was successfully able

Results

numbers between 0 and 100

■ Using Gray code instead of binary did not improve results

■ Perhaps needs a second hidden layer, or implement some

heuristics to reduce local minima issues

Cheung/Cannons 40

Neural Networks

Conclusion

neural networks and also demonstrated their

ability to learn the desired function and generalize

when needed

● The ability of neural networks to learn and

generalize in addition to their wide range of

applicability makes them very powerful tools

Cheung/Cannons 41

Neural Networks

Cheung/Cannons 42

Neural Networks

Acknowledgements

Council (NSERC)

● University of Manitoba

Cheung/Cannons 43

Neural Networks

References

[AbDo99] H. Abdi, D. Valentin, B. Edelman, Neural Networks, Thousand Oaks, CA: SAGE Publication

Inc., 1999.

[Hayk94] S. Haykin, Neural Networks, New York, NY: Nacmillan College Publishing Company, Inc., 1994.

[Mast93] T. Masters, Practial Neural Network Recipes in C++, Toronto, ON: Academic Press, Inc., 1993.

[Scha97] R. Schalkoff, Artificial Neural Networks, Toronto, ON: the McGraw-Hill Companies, Inc., 1997.

[WeKu91] S. M. Weiss and C. A. Kulikowski, Computer Systems That Learn, San Mateo, CA: Morgan

Kaufmann Publishers, Inc., 1991.

[Wass89] P. D. Wasserman, Neural Computing: Theory and Practice, New York, NY: Van Nostrand

Reinhold, 1989.

Cheung/Cannons 44

- Artificial Neural Networks (ANN)Uploaded byДушко Маџоски
- Neural NetworksUploaded byKiran Moy Mandal
- Neural NetworksUploaded byrian ngganden
- Neural NetworksUploaded bymundosss
- Chapter 6 - Feedforward Deep NetworksUploaded byjustdls
- Neural_Computing___An_Introduction.pdfUploaded byAjith Suresh
- NeuralNetworkFAQUploaded bykarvinu2020
- Multilayered Feed Forward Artificial Neural Network ModelUploaded byaman_hira9
- Investigation of Phytoplankton Dynamics Using Time Series Analysis of Biophysical Parameters in OssieUploaded byNisha Cruz
- medical image classificationUploaded byGaurav Jaiswal
- Proceedings_NSAIUploaded byMohan
- Forecasting the Tehran Stock Market by Artificial Neural NetworkUploaded byEditor IJACSA
- Sonar-Based Mobile Robot Navigation Through Supervised Learning on a Neural NetUploaded byjejudo
- Still an Alien Planet in Soft ComputingUploaded byRick Feed
- DL501 Course SummaryUploaded byKRISTOPHER_86
- Chapter 5 ANNUploaded byHafisIzran
- The Nature of Code - Chapter 10Uploaded byAnonymous vvSzEkh
- A Basic Introduction To Neural Networks.docUploaded byDennis Ebenezer Dhanaraj
- PFC Dynamic Neural Networks for MBS in Mechanical Systems.pdfUploaded byrgx112
- Geodezija_2012_04_Chapter6Uploaded byRamo Kiss
- Bucket ComputerUploaded bybrich1031
- A Cascade of Artificial Neural NetworksUploaded bydashan_r215
- Neural NetworksUploaded byFokak Menny
- 23.pdfUploaded bymsmsoft90
- A Trading System for FTSE-100 FuturesUploaded byMarcos Luis Almeida Barbosa
- Artificial Neural Network TermsUploaded byseph8765
- 07A7EC01-NEURALNETWORKSANDFUZZYLOGICUploaded bysriksjntu4508
- Evolution of Neural Net ArchitecturesUploaded byDavid Henriques
- Artificial Neural NetworksUploaded byJonathan Laurence
- Download(1)Uploaded bySURYA

- Field.serviceUploaded byjamie8889
- 8051 Instruction Set SummaryUploaded byanilshaw27
- GEK-114903 OC6000e Hardware Manual_V2.0Uploaded byMagoroku D. Yudho
- 6.QPSKUploaded bySai raghavendra
- Day1 2 AddOn OverviewUploaded byRajeev Singh
- Rigol GradientOne - One PagerUploaded byNick Edwards
- SUCESS FACTOR GUIDEUploaded bychintan mehta
- Case Study Test&ControlUploaded byAnkit
- Principles of Data CommunicationUploaded byMisha Echipare Antonio
- K-ROSET Aro Project Manual_EN.pdfUploaded bygendalf1
- ABAP Development Naming Standards.pdfUploaded byleeichang
- Oracle Fusion Cloud FinancialsUploaded byAkash
- Entity Relationship ModelUploaded byMalik Umair Alam
- Software Requirement Specification - Tariq KhanUploaded bysun.developer7227
- Principles of programming with C++ by Michael ObiriUploaded byMike
- Numerical(Lecture Notes)Uploaded byJeremy Borres
- in-10-12-2018-EXTREAMIPTVUploaded byVivek
- Rics 2011 BIM Survey ReportUploaded bypresario60
- opt_25491_2063846081Uploaded bydanielesmar
- DSS CHAP 10Uploaded byNau Ha'akili
- license.rtfUploaded byJoe Bledos
- National Conference KakinadaUploaded bydrvempada
- c++ (cpp) programs with numerical problemsUploaded byv2brother
- Npp ExecUploaded byJanki Desai
- Novo Documento de Texto 2.txtUploaded byramos
- cpl gatpUploaded byrsn_surya
- Summary of History of ComputersUploaded byHyung Bae
- Training_F4_V1.3.pdfUploaded bydie6o_br
- Alloy - A General-Purpose Higher-Order Relational Constraint SolverUploaded byLeonan Oliveira
- Source-to-Pay: Advancing from Pure Cost Optimization to Value GenerationUploaded byCognizant