Artificial Neural Networks (ANNs)

Perceptron
 Contents
 Introduction

 Perceptron

 Perceptron Convergence Theorem

 Error-correction learning algorithm

 Introduction
In the formative years of neural networks (1943–1958), several researchers stand out for their pioneering
contributions:

• McCulloch and Pitts (1943) for introducing the idea of neural networks as computing machines. They
presented the McCulloch-pitts model of neuron which is a neuron with a threshold activation function
(signum function).

• Hebb (1949) for postulating the first rule for self-organized learning.

• Rosenblatt (1958) for proposing the perceptron as the first model for learning with a teacher (i.e.,
supervised learning).
 Perceptron
 The perceptron is the simplest form of a neural network used for the classification of patterns said to be
linearly separable (i.e., patterns that lie on opposite sides of a hyperplane).

 Basically, it consists of a single nonlinear neuron (McCulloch-pitts model of neuron) with adjustable
synaptic weights and bias. It consists of a linear combiner followed by a hard limiter that performs the
signum function. Accordingly, the neuron produces an output equal to +1 if the hard limiter input is
positive, and -1 if it is negative.

 The perceptron built around a single neuron is limited to

performing pattern classification with only two classes
(hypotheses). By expanding the output (computation) layer
of the perceptron to include more than one neuron,
we may correspondingly perform classification with more than
two classes. However, the classes have to be linearly separable
for the perceptron to work properly.
 Perceptron Convergence Theorem
 Theorem

if the patterns (vectors) used to train the perceptron are drawn from two linearly separable classes, then the
perceptron algorithm converges and positions the decision surface in the form of a hyperplane between the two
classes.
 Perceptron Convergence Theorem

Perceptron forward computation:

1. Compute the net input (v), which is the linear combiner of input signals (x) multiplied by their
corresponding weights (w)

2. Apply activation function to net input (v) to get the final output (y)

The goal of the perceptron is to correctly classify the set of externally

applied stimuli x1, x2, ..., xm into one of two classes, L1 or L2.
The decision rule for the classification is to assign the point
represented by the inputs x1, x2, ..., xm to class L1 if the perceptron
Output y is +1 and to class L2 if it is -1.
 Perceptron Convergence Theorem

The perceptron needs to position the decision surface in the form of a hyperplane between the two classes L1
and L2. the equation of the decision surface is given by:

For example, if we only have 2 input signals x1 and x2, then in this case
the decision surface is a straight line and its equation is given by:

The learning process of the perceptron is to adapt the synaptic weights in

iteration-by-iteration until we find the weights that define the decision
Surface that separate the two classes L1 and L2.
 Error-correction Learning Algorithm
The Derivation of Error-Correction learning Algorithm (update weight equation):

 The Input fed into the perceptron at time n is (m+1)-by-1 vector

 The weight is also (m+1)-by-1 vector

 The linear combiner (net input)

 The final output

𝒚 (𝒏)=𝝋 (𝒗 (𝒏))
 Error-correction Learning Algorithm
 As we said, the perceptron classifies two linearly separable classes L1 and L2. Let’s draw a subset of patterns
H1 from class L1 and a subset H2 from class L2 to train the perceptron. The union of H1 and H2 is the
complete training set H.

 The classification rule : is to assign the pattern (vector) drawn from the training set H to class L1 if y=+1 and
to class L2 if y=-1

 The training process involves the adjustment of the weight vector w in such a way that the two classes are
linearly separable. That is, there exists a weight vector w such that we may state

 The training problem for the perceptron is then to find a weight vector w such that the two inequalities are
satisfied.
Error-correction Learning Algorithm
Error-correction Learning Algorithm