4 views

Uploaded by Duc DQ

Backprop

- Alignment Machine Learning
- A Rubaai
- Application Determination
- Burr
- Neural Network
- PERFORMANCE ANALYSIS OF WAVELET & BLUR INVARIANTS FOR CLASSIFICATION OF AFFINE AND BLURRY IMAGES
- 558-214
- Recognition of Printed Bangla Document from Textual Image Using Multi-Layer Perceptron (MLP) Neural Network
- Training ANN Hybrid PSO-CS Algorithm
- 01 Overview
- USAID Report_Artificial Intelligence in Global Health (May 2019 report)
- 09 Class Advanced
- 40 Amit Goyal
- 11_chapter%203[1].pdf
- Development of Adaptive Neuro Fuzzy Inference System for Estimation of Evapotranspiration
- 3
- The-State-of-Machine-Learning-and-Artificial-Intelligence[1]
- AWS Certified Machine Learning - Specialty_Exam Guide (1)
- Ieee Paper Destilacion Redes Neuronales
- IRJET-Optimization of CI Engine Performance Parameters for Jatropha Biodiesel Blending Fuel by Using ANN Software

You are on page 1of 28

Backpropagation

Chris Marriott

Ryan Shirley

CJ Baker

Thomas Tannahill

Agenda

Backpropagation: The Math

Advantages and Disadvantages of Gradient

Descent and other algorithms

Enhancements of Gradient Descent

Other ways of minimizing error

Review

the human brain

Nodes created as an analog to neurons

Mainly used for classification problems (i.e.

character recognition, voice recognition,

medical applications, etc.)

Review

values, activation function, and an output

Weighted inputs

Output

Activationfunction=f( (inputs*weight))

Review

4InputAND

Inputs

Threshold=1.5

Outputs

Threshold=1.5

Inputs

Threshold=1.5

Allweights=1andalloutputs=1ifactive0otherwise

Review

Input1

(1,1)

(0,1)

1.5=w1*I1+w2*I2

(0,0)

(1,0)

Input2

Review

Demonstrates need for hidden layer

Input1

(0,1)

(1,1)

Input2

(0,0)

(1,0)

OutputLayer

0

X0,0

HiddenLayer

1

W1,0

W0,0

Wi,0

i

InputLayer

9

X9,0

X1,0

Backpropagation

Backpropagation

Backpropagation

Calculation of error

k = f(Dk) -f(Ok)

Backpropagation

Backpropagation

Learning Rate

(eg. 0.25)

Error at k

Backpropagation

Descent

Advantages

Standard method and generally works well

Disadvantages

Can get stuck in local minima resulting in suboptimal solutions

Local Minima

Local

Minimum

GlobalMinimum

Simulated Annealing

Advantages

Disadvantages

Much more complicated implementation

Advantages

Less likely to get stuck in local minima

Disadvantages

Memory intensive for large nets

Simplex Algorithm

Advantages

Easy to implement

Disadvantages

Enhancements To Gradient

Descent

Momentum

current movement

Enhancements To Gradient

Descent

Momentum

Often finds a minimum in less steps

w(t) = -n*d*y + a*w(t-1)

n is the learning rate

d is the error

y is different depending on which layer we are calculating

a is the momentum parameter

Enhancements To Gradient

Descent

That learning rate is determined by the sign of the gradient

of the error function from the last iteration

the learning rate is increased

The signs are more likely to differ on a steep slope so the

learning rate is decreased

slopes

Enhancements To Gradient

Descent

Adaptive Backpropagation

Possible Problems:

the overall error may increase

Solution:

and if it is greater than the previous error reject that

adaptation and calculate new learning rates

Enhancements To Gradient

Descent

Uses adaptive method and momentum so long as the sign

of the gradient does not change

traversal of gradual slopes

will cancel the drastic drop in learning rate

This allows for the function to roll up the other side of the

minimum possibly escaping local minima

Enhancements To Gradient

Descent

SuperSAB

faster than gradient descent

Overall this algorithm is less sensitive (and so is less

likely to get caught in local minima)

Randomly select from input classes

Randomly change value of input node (with low

probability)

loss in generalization

Removing neurons has the opposite effect

Resources

Henseler

Artificial Intelligence: A Modern Approach, S.

Russell & P. Norvig

501 notes, J.R. Parker

www.dontveter.com/bpr/bpr.html

www.dse.doc.ic.ac.uk/~nd/surprise_96/journal/

vl4/cs11/report.html

- Alignment Machine LearningUploaded byCharbel Avognon
- A RubaaiUploaded bysherykhann
- Application DeterminationUploaded bycraig
- BurrUploaded bynanoteraCH
- Neural NetworkUploaded bySaranya Ramaswamy
- PERFORMANCE ANALYSIS OF WAVELET & BLUR INVARIANTS FOR CLASSIFICATION OF AFFINE AND BLURRY IMAGESUploaded bygreataks
- 558-214Uploaded byJorge Sandoval Juarez
- Recognition of Printed Bangla Document from Textual Image Using Multi-Layer Perceptron (MLP) Neural NetworkUploaded byijcsis
- Training ANN Hybrid PSO-CS AlgorithmUploaded bySaravjeet Singh
- 01 OverviewUploaded bywerwerwer
- USAID Report_Artificial Intelligence in Global Health (May 2019 report)Uploaded bywaqas724
- 09 Class AdvancedUploaded byENNAJAR
- 40 Amit GoyalUploaded byAgung Arif Nur Wibowo
- 11_chapter%203[1].pdfUploaded byAnonymous 1nUCb4Fj9
- Development of Adaptive Neuro Fuzzy Inference System for Estimation of EvapotranspirationUploaded byInternational Journal for Scientific Research and Development - IJSRD
- 3Uploaded byGaurav SK
- The-State-of-Machine-Learning-and-Artificial-Intelligence[1]Uploaded byBiomodelling
- AWS Certified Machine Learning - Specialty_Exam Guide (1)Uploaded bykaushal
- Ieee Paper Destilacion Redes NeuronalesUploaded byixmitlopez
- IRJET-Optimization of CI Engine Performance Parameters for Jatropha Biodiesel Blending Fuel by Using ANN SoftwareUploaded byIRJET Journal
- Artificial IntelligenceUploaded byAgalya Ganesh
- Bridging the skills gapUploaded byAnonymous iNxLvw
- Pattern classification manualUploaded byAlexander Jagannathan
- ALTA_2015_December.pdfUploaded byfestival val
- Analyst SkillsUploaded byGarvit Harisinghani
- AWS Certified Machine Learning - Specialty_Exam Guide.pdfUploaded bykaushal
- 205-WXES1110-L-38200916439Uploaded bymurnianashazwen
- Artificial IntelligenceUploaded byGeo
- Recognition of Characters and Patterns using Neural NetworkUploaded byEditor IJRITCC
- AWS Certified Machine Learning - Specialty_Exam Guide (1).pdfUploaded bykaushal

- 1. a Knowledge Management Framework in Software Requirements Engineering Based on the SECI ModelUploaded byndraa_scout124
- report1Multi-ObjectiveOptimizationUploaded byVladimir Garcia Quintana
- Complexity MapUploaded byCoZuK
- The Why What Who When and How of Software RequirementsUploaded bytrimitu
- Hybrid Ant Colony Optimization for Real-World Delivery Problems Based on Real Time and Predicted Traffic in Wide Area Road NetworkUploaded byCS & IT
- Design of a PID Controller using VHDLUploaded byIRJET Journal
- 2 4 3 pringles experiment folioUploaded byapi-272055202
- A Review on Implementation of TPM in Manufacturing IndustryUploaded byIJMER
- Lecture 1 (Introduction-Homeostasis)Uploaded byHidayah Hasan
- Procedure for Designing and Developing ProjectUploaded byMalu Paras Lacson
- Abet Syllabus FormatUploaded byfcb1138
- Lec 10Uploaded byKaran Chawla
- Proof Test Procedure Effectiveness on Safety Instrumented SystemsUploaded byNaresh Bajaj
- 1. Questions & Answers on Mathematical Models of Physical SystemsUploaded bykibrom atsbha
- MKT110009Uploaded byAmit Jha
- Open Pit Mine Planning With Blending ConstraintsUploaded byJose Gonzales
- Functional Specification - Document ExampleUploaded byJohn Wesely
- The Evolution of Systems Analysis and Control a Personal PerspectiveUploaded byTonny Yu
- Short Note on Pid and Plc_2Uploaded byRadab Hameed
- UML 2.0 TutorialUploaded byMisael Fernando Perilla Benitez
- LeaUploaded byazariel.rodrigo
- DISSIPATIVE INDIVIDUATION Esra AtamerUploaded byJamie Allen
- ID Ppt_Block 1_Unit 3-2Uploaded by黃達明
- MMP2Uploaded byatef shaar
- System 800xA - The Power of AutomationUploaded byManuel Penafiel
- Extreme ProgrammingUploaded byM Sin Gh
- FINAL REPORT.pdfUploaded byMuhammad Aizat
- Matematika Optimisasi -1Uploaded byFindar
- Lecture 1Uploaded byNikhil Minhas
- 13-rt-locusUploaded byHimanshu Jain