Backpropagation

Back-propagation Neural
Network
• Back Propagation described by Arthur E. Bryson and Yu-Chi Ho in
1969
• The term is an abbreviation for backwards propagation of errors,
the errors (and therefore the learning) propagate backwards from
the output nodes to the inner nodes.
• Backpropagation is used to calculate the gradient of the error of the
network with respect to the network's modifiable weights.
• This gradient is almost always then used in a simple stochastic
gradient descent algorithm to find weights that minimize the error.
• Backpropagation usually allows quick convergence on satisfactory
local minima for error in the kind of networks to which it is suited.
 Back-propagation Neural
Network
x1 x2 t
0 0 1
0 1 0
1 0 0
1 1 1

W11 0.3 W14 0.6

W12 0.4 W15 0.6
W13 0.5 W24 0.6
W21 0.3 W25 0.6
W22 0.4 W34 0.6 W46 0.3
W23 0.5 W35 0.6 W56 0.5
 Back-propagation Neural
Network
Net 1  𝑦 1= 1
1+ 𝑒− 𝑛𝑒𝑡 ( 𝜆 )

W11 0.3 W14 0.6

W12 0.4 W15 0.6
W13 0.5 W24 0.6
W21 0.3 W25 0.6
W22 0.4 W34 0.6 W46 0.3
W23 0.5 W35 0.6 W56 0.5
 Back-propagation Neural
Network
Net 2  𝑦 2= 1
1+ 𝑒− 𝑛𝑒𝑡 (𝜆 )

W11 0.3 W14 0.6

W12 0.4 W15 0.6
W13 0.5 W24 0.6
W21 0.3 W25 0.6
W22 0.4 W34 0.6 W46 0.3
W23 0.5 W35 0.6 W56 0.5
 Back-propagation Neural
Network
Net 3  𝑦 3= 1
− 𝑛𝑒𝑡 (𝜆 )
1+ 𝑒

W11 0.3 W14 0.6

W12 0.4 W15 0.6
W13 0.5 W24 0.6
W21 0.3 W25 0.6
W22 0.4 W34 0.6 W46 0.3
W23 0.5 W35 0.6 W56 0.5
 y1 0.5
y2 0.5

Back-propagation Neural y3 0.5

Network
Net 4 Net4 = w14y1 +w24y2 + w34y3
= 0.6*0.5 + 0.6*0.5 + 0.6*0.5
= 0.3 + 0.3 + 0.3
= 0.9

W11 0.3 W14 0.6

W12 0.4 W15 0.6
W13 0.5 W24 0.6
W21 0.3 W25 0.6
W22 0.4 W34 0.6 W46 0.3
W23 0.5 W35 0.6 W56 0.5
 y1 0.5
y2 0.5

Back-propagation Neural y3
y4
0.5
0.71

Network
Net 5 Net5 = w15y1 +w25y2 + w35y3
= 0.6*0.5 + 0.6*0.5 + 0.6*0.5
= 0.3 + 0.3 + 0.3
= 0.9

W11 0.3 W14 0.6

W12 0.4 W15 0.6
W13 0.5 W24 0.6
W21 0.3 W25 0.6
W22 0.4 W34 0.6 W46 0.3
W23 0.5 W35 0.6 W56 0.5
 y1 0.5
y2 0.5

Back-propagation Neural y3
y4 0.71
0.5

Network y5 0.71

Net6 = w46y4 +w56y5

Net 6 = 0.3 * 0.71 + 0.5 * 0.71
= 0.21 + 0.35
= 0.56

W11 0.3 W14 0.6

W12 0.4 W15 0.6
W13 0.5 W24 0.6
W21 0.3 W25 0.6
W22 0.4 W34 0.6 W46 0.3
W23 0.5 W35 0.6 W56 0.5
 y1 0.5
y2 0.5

Back-propagation Neural y3
y4
0.5
0.71

Network y5
y6
0.71
0.63

δ6 = Y6 (1 - Y6)(t- Y6) δ6 = Y6 (1 - Y6)(t- Y6)

= 0.63 (1-0.63)(1-0.63)
t = 0.63*0.37*0.37
Y = 0.086

W11 0.3 W14 0.6

W12 0.4 W15 0.6 x1 x2 t
W13 0.5 W24 0.6 0 0 1
W21 0.3 W25 0.6 0 1 0
W22 0.4 W34 0.6 W46 0.3 1 0 0
W23 0.5 W35 0.6 W56 0.5 1 1 1
δ6 0.086 y1 0.5
y2 0.5

Back-propagation Neural y3
y4
0.5
0.71

Network y5
y6
0.71
0.63
 5  Y 5(1  Y 5)   6W 56 δ5 = Y5 (1 – Y5) ∑ δ6 W56
= 0.71 (1-0.71)(0.086 * 0.5)
= 0.71 * 0.29 * 0.043
= 0.0088

W11 0.3 W14 0.6

W12 0.4 W15 0.6
W13 0.5 W24 0.6
W21 0.3 W25 0.6
W22 0.4 W34 0.6 W46 0.3
W23 0.5 W35 0.6 W56 0.5
δ6 0.086 y1 0.5
δ5 0.0088 y2 0.5

Back-propagation Neural y3
y4
0.5
0.71

Network y5
y6
0.71
0.63
 4  Y 4(1  Y 4)   6W 46 δ4 = Y4 (1 – Y4) ∑ δ6 W46
= 0.71 (1-0.71)(0.086 * 0.3)
= 0.71 * 0.29 * 0.0258
= 0.0053

W11 0.3 W14 0.6

W12 0.4 W15 0.6
W13 0.5 W24 0.6
W21 0.3 W25 0.6
W22 0.4 W34 0.6 W46 0.3
W23 0.5 W35 0.6 W56 0.5
δ6 0.086 y1 0.5
δ5 0.0088 y2 0.5
δ4 0.0053
Back-propagation Neural y3
y4
0.5
0.71

Network y5
y6
0.71
0.63
 3  Y 3(1  Y 3) ( 4W 34,  5W 35) δ3=0.5*0.5(0.0053*0.6+0.0088*0.
6)
= 0.5 * 0.5 * (0.0032+0.0053)
= 0.5 * 0.5 * 0.0085
= 0.0021

W11 0.3 W14 0.6

W12 0.4 W15 0.6
W13 0.5 W24 0.6
W21 0.3 W25 0.6
W22 0.4 W34 0.6 W46 0.3
W23 0.5 W35 0.6 W56 0.5
δ6 0.086 y1 0.5
δ5 0.0088 y2 0.5
δ4
δ3
0.0053
0.0021
Back-propagation Neural y3
y4
0.5
0.71

Network y5
y6
0.71
0.63
 2  Y 2(1  Y 2) ( 4W 24,  5W 25) δ2=0.5*0.5(0.0053*0.6+0.0088*0.
6)
= 0.5 * 0.5 * (0.0032+0.0053)
= 0.5 * 0.5 * 0.0085
= 0.0021

W11 0.3 W14 0.6

W12 0.4 W15 0.6
W13 0.5 W24 0.6
W21 0.3 W25 0.6
W22 0.4 W34 0.6 W46 0.3
W23 0.5 W35 0.6 W56 0.5
δ6 0.086 y1 0.5
δ5 0.0088 y2 0.5
δ4
δ3
0.0053
0.0021
Back-propagation Neural y3
y4
0.5
0.71
δ2 0.0021 Network y5
y6
0.71
0.63
 1  Y 1(1  Y 1) ( 4W 14,  5W 15) δ1=0.5*0.5(0.0053*0.6+0.0088*0.
6)
= 0.5 * 0.5 * (0.0032+0.0053)
= 0.5 * 0.5 * 0.0085
= 0.0021

W11 0.3 W14 0.6

W12 0.4 W15 0.6
W13 0.5 W24 0.6
W21 0.3 W25 0.6
W22 0.4 W34 0.6 W46 0.3
W23 0.5 W35 0.6 W56 0.5
δ6 0.086 y1 0.5
δ5 0.0088 y2 0.5
δ4
δ3
0.0053
0.0021
Back-propagation Neural y3
y4
0.5
0.71
δ2 0.0021 Network y5
y6
0.71
0.63
δ1 0.0021

W’11= W11+ α * δ1 * x1
= 0.3 + 0.1 * 0.0021 * 0
= 0.3

W’21= W21+ α * δ1 * x2
= 0.3 + 0.1 * 0.0021 * 0
= 0.3
W11 0.3 W14 0.6
W12 0.4 W15 0.6
W13 0.5 W24 0.6
W21 0.3 W25 0.6
W22 0.4 W34 0.6 W46 0.3
W23 0.5 W35 0.6 W56 0.5
δ6 0.086 y1 0.5
δ5 0.0088 y2 0.5
δ4
δ3
0.0053
0.0021
Back-propagation Neural y3
y4
0.5
0.71
δ2 0.0021 Network y5
y6
0.71
0.63
δ1 0.0021

W’12= W12+ α*δ2*x1

= 0.4 + 0.1 * 0.0021 * 0
= 0.4

W’22= W22+ α*δ2*x2

= 0.4 + 0.1 * 0.0021 * 0
= 0.4
W11 0.3 W14 0.6
W12 0.4 W15 0.6
W13 0.5 W24 0.6
W21 0.3 W25 0.6
W22 0.4 W34 0.6 W46 0.3
W23 0.5 W35 0.6 W56 0.5
δ6 0.086 y1 0.5
δ5 0.0088 y2 0.5
δ4
δ3
0.0053
0.0021
Back-propagation Neural y3
y4
0.5
0.71
δ2 0.0021 Network y5
y6
0.71
0.63
δ1 0.0021

W’13= W13+ α*δ3*x1

= 0.5 + 0.1 * 0.0021 * 0
= 0.5

W’23= W23+ α*δ3*x2

= 0.5 + 0.1 * 0.0021 * 0
= 0.5
W11 0.3 W14 0.6
W12 0.4 W15 0.6
W13 0.5 W24 0.6
W21 0.3 W25 0.6
W22 0.4 W34 0.6 W46 0.3
W23 0.5 W35 0.6 W56 0.5
δ6 0.086 y1 0.5
δ5 0.0088 y2 0.5
δ4
δ3
0.0053
0.0021
Back-propagation Neural y3
y4
0.5
0.71
δ2 0.0021 Network y5
y6
0.71
0.63
δ1 0.0021

W’14= W14+ α*δ4*y1

= 0.6 + 0.1 * 0.0053 * 0.5
= 0.600265
= 0.6

W’24= W24+ α*δ4*y2

= 0.6 + 0.1 * 0.0053 * 0.5
W11 0.3 W14 0.6 = 0.6
W12 0.4 W15 0.6
W13 0.5 W24 0.6 W’34= W34+ α*δ4*y3
W21 0.3 W25 0.6
= 0.6 + 0.1 * 0.0053 * 0.5
W22 0.4 W34 0.6 W46 0.3
= 0.6
W23 0.5 W35 0.6 W56 0.5
δ6 0.086 y1 0.5
δ5 0.0088 y2 0.5
δ4
δ3
0.0053
0.0021
Back-propagation Neural y3
y4
0.5
0.71
δ2 0.0021 Network y5
y6
0.71
0.63
δ1 0.0021

W’15= W15+ α*δ5*y1

= 0.6 + 0.1 * 0.0021 * 0.5
= 0.6

W’25= W25+ α*δ5*y2

= 0.6 + 0.1 * 0.0021 * 0.5
= 0.6
W11 0.3 W14 0.6
W12 0.4 W15 0.6 W’35= W35+ α*δ5*y3
W13 0.5 W24 0.6
= 0.6 + 0.1 * 0.0021 * 0.5
W21 0.3 W25 0.6
= 0.6
W22 0.4 W34 0.6 W46 0.3
W23 0.5 W35 0.6 W56 0.5
δ6 0.086 y1 0.5
δ5 0.0088 y2 0.5
δ4
δ3
0.0053
0.0021
Back-propagation Neural y3
y4
0.5
0.71
δ2 0.0021 Network y5
y6
0.71
0.63
δ1 0.0021

W’46= W46+ α*δ6*y4

= 0.3 + 0.1 * 0.086 * 0.71
= 0.31

W’56= W46+ α*δ6*y5

= 0.5 + 0.1 * 0.086 * 0.71
= 0.51
W11 0.3 W14 0.6
W12 0.4 W15 0.6
W13 0.5 W24 0.6
W21 0.3 W25 0.6
W22 0.4 W34 0.6 W46 0.3
W23 0.5 W35 0.6 W56 0.5