Back Propagation Learning Algorithm

What do each layers do?

1st layer draws linear boundaries 2ND layer combines the boundaries

3rd layer generate arbitrarily complex boundaries

Back Propagation
BP has two phases
Forward pass phase
Computes functional signal Feed forward propagation of input pattern signals through network

Backward pass phase

Computes error signal Propagates the error backward through network starting at output units ( where the error is the difference between actual and desired output values)

An Overview of training
The objective of training network is to adjust the weights so that application of a set of inputs produces the desired set of outputs Training assumes that each input vector is paired with a target vector representing the desired output ; together are these are called a training pair The group of training pair is called a training set

Training Algorithm
1.
2. 3. Forward pass Select the next training pair from the training set. Apply the input vector to the new network input Calculate the output of the network Calculate the error between actual output and desired output
Backward Pass

4. 5.

Adjust the weight of the network in a way that it minimizes the error Repeat step 1 to 4 for each vector in the training set until the error for the entire set is acceptably low

Forward Pass
Signals propagates from the network input to the output The input target vector pair X and T become the training set Input is applied at the front layer and output NET is calculated in each layer as the weighted sum of its neurons inputs
NET = XW

The activation function squashes NET to produce OUT for each neuron in that layer Once the set of outputs for a layer is found, it serves as the input to the next layer The process is repeated, layer by layer, until the final set of output is produced
OUT = F ( XW )

Backward Pass
Adjusting the weights of the output layer Consider a neuron q in the output layer (last) and training process of a single weight from p in the hidden layer j to q in output layer k
Target

Backward Pass
The output of a neuron in layer k is subtracted from its target value to produce an ERROR signal . This is multiplied by derivative of squashing function [OUT ( 1- OUT ) ] calculated for that layer neuron k thereby producing the value = OUT (1- OUT) (Target-OUT) This is multiplied by OUT from a neuron (p,j) i.e. the output of previous layer

q,k * OUT p,j

It is in turn multiplied by a learning rate coefficient (typically 0.01 to 1.0 ) and the result is added to the weight
pq,k = q,k * OUT p,j

1
2

pq,k( t + 1) = pq,k( t ) + pq,k

An identical process is performed for each weight proceeding from a neuron in the hidden layer to a neuron in the output layer

Backward Pass (Contd)

Where

pq,k( t)

= the value of a weight from neuron p in the hidden layer to neuron q in the output layer at step t ( before adjustment); note that the subscript k indicates that the weight is associated with its destination layer
adjustment)

pq,k( t + 1)= value of the weight at step t+1 (after

q,k
output layer k
OUT p,j = the value of OUT for neuron p in the hidden layer j = the value of for neuron q in the

Adjusting the weight in the hidden layers

Adjusting the weight in the hidden layers

Hidden layer have no target vector Back propagation trains the hidden layer by propagating the output error back through the network layer by layer, adjusting weights at each layer The equation 1 & 2 are used for all layers both output and hidden But for hidden layer calculation of is different from that of output layer is calculated for each of the neuron in the output layer It is used to adjust the weights feeding into the output layer, then it is propagated back through the same weights to generate a value for each neuron in the first hidden layer These values of are in turn used to adjust the weights of this hidden layer and in a similar way are propagated back to all the preceding layers

Adjusting the weight in the hidden layers

Consider a single neuron p in the hidden layer just before output layer In the forward pass this neuron propagates its output value to neurons in the output layer through the interconnecting weights During training these weights operate in reverse, passing the value of from the output layer back to the hidden layer Each of these weights is multiplied by the value of the neuron to which it connects in the output layer. The value of needed for the hidden - layer neuron is produced by summing all such products and multiplied by the derivative of the squashing function

p,j = OUTp,j ( 1- OUTp,j ) ( q,k pq,k )

With in hand the weights feeding the first layer can be adjusted using equations 1 and 2

Adjusting the weight in the hidden layers

The set of s in the output layer can be represented as Dk. The set of weights for the output layer is the array Wk. Now we want to arrive at Dj(vector for hidden layer) can be calculated using 2 steps. Dj=Dk Wk $[ Dj $(I - Oj) ] --------------(A)

Adjusting the weight in the hidden layers

1)Multiply the vector of the o/p layer by transpose of the weight matrix connecting to the o/p layer. 2) Multiply each component of the resulting product by derivative of the squashing function for the corresponding neuron in the hidden layer Eqn: (A) $is said to denote component by component multiplication Oj is the o/p vector of layer j i is a vector whose components are all 1s.