Neural Network

Module 1
Contents
Introduction: Biological Neuron – Day 1
Artificial Neural Model – Types of activation functions – Day 2
Architecture: Feedforward and Feedback –Day 3
Convex Sets, Convex Hull and Linear Separability – Day 3
Non-Linear Separable Problem – Day 4
XOR Problem – Day 4
Multilayer Networks –Day 5
Learning: Learning Algorithms – Day 5
Error correction and Gradient Descent Rules – Day 6
Learning objective of TLNs – Day 6
Perceptron Learning Algorithm – Day 7
Perceptron Convergence Theorem – Day 8

28-10-2021 Neural Network : Module 1 2

Introduction: Biological Neuron

Structure of a Biological Neuron

• The information-processing cells of the brain are the neurons.
• Each neuron has a soma or cell body which contains the cell's nucleus and other vital components called
organelles which perform specialized tasks.
• There are also its main communication links:
1. A set of dendrites which form a tree-like structure that spreads out from the cell. The neuron receives
its input electrical signals along these.
2. A single axon which is a tubular extension from the cell soma that carries an electrical signal away
from the soma to another neuron for processing.
• The dendrites and axon together are sometimes called processes of the cell.

28-10-2021 Neural Network : Module 1 3

Neuron

28-10-2021 Neural Network : Module 1 4

Glial Cells and Membrane
• Accounts for one-tenth of the cells in the
brain.
• The three varieties:
• Astroglia : Surround the neurons and isolate them
from the smallest blood vessels of the brain.
• Microglia : Move continuously between neurons
and glia to clean up debris.
• Oligodendroglia : Send out membranous processes
that wrap themselves tightly around axon, forming a
dense layer of spiralling membrane called myelin
sheath.
• The plasma membrane that encloses the
neuron is a two layered structure about 90 Å
thick.

28-10-2021 Neural Network : Module 1 5

Cell Membrane
• The two layers of plasma membrane consist of phospholipid molecules.
• Each molecule has a hydrophilic head and a hydrophobic tail made up of hydrocarbon chain.
• Proteins regulate the transport of ions through pores in the membrane and move ions across the
membrane from one side to the other hence called as ion channels.
• The properties of ion channels are :
• Allow the passage of ion at a very high rate.
• Very selective to a specific ion.
• Open or close depending upon a voltage or chemical signal.

28-10-2021 Neural Network : Module 1 6

Resting Potential and Nernst Equation
• Resting Membrane Potential is the difference in electrical potential across the membrane due to
the difference in ion concentration inside and outside the cell.

28-10-2021 Neural Network : Module 1 7

•

28-10-2021 Neural Network : Module 1 8

 Ion Concentration Concentration
Inside (mM) outside(mM)
15 150 • R = 8.31432
×103 N⋅m⋅kmol−1⋅K−1
100 5
• T = 37 kelvin
13 150 • F = 96,485 C/mol

28-10-2021 Neural Network : Module 1 9

Graded Potential Spread in Space and Time
• External signals impinging on the neuron at
synapses which presynaptic neurons make at
dendritic spines create disturbance in the cell
potential called graded potential.
• When an impinging signal decreases the
internal potential below the resting potential
then the neuron is said to be hyperpolarized.
• If the external disturbance pushes the potential
up in the positive direction with respect to the
resting potential, then the neuron is said to be
hypo polarized.

28-10-2021 Neural Network : Module 1 10

Action Potential
• Neurons receive inputs from other neurons along their dendrites at points of
contact called synapses.
• The inputs take the form of small electrical disturbances called postsynaptic
potentials(PSP).
• The cell soma receive these small disturbances which are superimposed upon each
other, reflecting a temporal integration of these potential.
• Ion channels exist in high density at the point where the axon meets the cell body
expanding into a structure called axon hillock.
• The ion channel constantly monitors the soma potential in such a way that when
the cell potential exceeds a threshold value, the neuron fires an action potential
that is transmitted down its axon towards a synaptic terminal.

28-10-2021 Neural Network : Module 1 11

 1. Stimulus starts the rapid change in voltage or action
potential. In patch-clamp mode, sufficient current
must be administered to the cell in order to raise the
voltage above the threshold voltage to start
membrane depolarization.
2. Depolarization is caused by a rapid rise in membrane
potential opening of sodium channels in the cellular
membrane, resulting in a large influx of sodium ions.
3. Membrane Repolarization results from rapid sodium
channel inactivation as well as a large efflux of
potassium ions resulting from activated potassium
channels.
4. Hyperpolarization is a lowered membrane potential
caused by the efflux of potassium ions and closing of
the potassium channels.
5. Resting state is when membrane potential returns to
the resting voltage that occurred before the stimulus
occurred.

28-10-2021 Neural Network : Module 1 12

Refractory Period
• The action potential cannot be regenerated
during the action potential.
• The action potential exist until the overshoot
phase of the action potential is complete and
the sodium channel closes.
• The period of in excitability lasts for 1-2ms
called as absolute refractory period.
• Absolute refractory period corresponds to
depolarization and repolarization,
whereas relative refractory
period corresponds to hyperpolarization.

28-10-2021 Neural Network : Module 1 13

Propagation of Action Potential

28-10-2021 Neural Network : Module 1 14

• Any disturbance in the equilibrium potential is conducted immediately to its
adjacent region in the form of a graded potential.
• The graded potential decays exponentially in space along the axon length.
• As the action potential waveform rises towards its peak at the axon hillock, the
potential along a considerable length of axonal membrane adjacent to the axon
hillock is also pushed above the threshold.
• This causes that region`s sodium channel to open, thereby initiating the entire
sequence of events that generate an action potential.
• The action potential is a self regenerating waveform that can travel large axonal
distances without any attenuation.

28-10-2021 Neural Network : Module 1 15

• As the action potential passes down a patch of axon, sodium-potassium pumps in that region begin
to restore the equilibrium ion concentration.
• The speed of conduction of an action potential is limited by the resistance to the flow of ionic
inside the axon which in turn depends on the diameter of the axon.
• The continuous retriggering of the action potential along the adjacent region short circuits the
graded potential- this needs to be prevented.
• Solution : Myelin sheath plays an important role.
• Myelin acts as an insulator and prevents the action potential from taking place by preventing the
influx and efflux of ions.
• The graded potential decays slightly before it is exposed at the node of Ranvier.
• This kind of conduction is called saltatory conduction.

28-10-2021 Neural Network : Module 1 16

Notation
•

28-10-2021 Neural Network : Module 1 17

Mathematical Symbols
•

28-10-2021 Neural Network : Module 1 18

Neuron Activation
•

28-10-2021 Neural Network : Module 1 19

Activation Functions
• Binary threshold
• Linear threshold
• Sigmoidal
• Gaussian
• Probabilistic

28-10-2021 Neural Network : Module 1 20

Binary Threshold Signal Function
•

28-10-2021 Neural Network : Module 1 21

•

28-10-2021 Neural Network : Module 1 22

Interpretation of Bias
•
Activation Signal

28-10-2021 Neural Network : Module 1 23

28-10-2021 Neural Network : Module 1 24
Linear threshold (Ramp) Signal Function
• The simplest function is the linear function where we have

∙ Such a signal function is unbounded.

∙ The bounded version of the linear signal function is the linear threshold function shown in the figure

28-10-2021 Neural Network : Module 1 25

28-10-2021 Neural Network : Module 1 26
•

28-10-2021 Neural Network : Module 1 27

•

28-10-2021 Neural Network : Module 1 28

Sigmoidal Signal Function

28-10-2021 Neural Network : Module 1 29

28-10-2021 Neural Network : Module 1 30
Gaussian Signal Function

28-10-2021 Neural Network : Module 1 31

28-10-2021 Neural Network : Module 1 32
Stochastic Neurons

∙ All the neuron signal functions introduced thus far are deterministic in the sense that the signal
value is completely determined by the instantaneous activation value that the neuron acquires, in
conjunction with the neuron signal function. There is no randomness.
∙ However, synaptic transmission in biological neurons is essentially a noisy process brought about
through random fluctuations resulting from the release of neurotransmitters, and numerous other
uncertain causes.
∙ such randomness generated through synaptic noise in a mathematically tractable framework
should be taken into account.
∙ Stochasticity into the neuron can be introduced by assuming that its activation to signal updates is
no longer deterministic.

28-10-2021 Neural Network : Module 1 33

•

28-10-2021 Neural Network : Module 1 34

•

28-10-2021 Neural Network : Module 1 35

28-10-2021 Neural Network : Module 1 36
ARCHITECTURES: FEEDFORWARD
AND FEEDBACK
∙ A neural network N can be viewed as a weighted directed graph in which artificial neurons are
nodes and directed weighted edges represent connections between neurons.
∙ Local groups of neurons can be connected either in
o a feedforward architecture, in which the network has no loops; or
o a feedback (recurrent) architecture, in which loops occur in the network because of feedback
connections.

28-10-2021 Neural Network : Module 1 37

Neural Network Generate Mappings
∙ The neurons in an artificial neural network are generally organized into layers.
∙ Inputs to the network are presented to the input layer; outputs are generated as signals of the output layer.
∙ Since the numeric activations and signals comprise vectors, it can be said that a input vector applied to the
input layer generates an output signal vector across the output layer.
∙ The signals may pass through one or more intermediate or hidden layers which transform the signals
depending on the neuron signal functions.
∙ The network performs a transformation linear or non-linear-from the input space to the output space.

∙ For example, if a 5-neuron input feeds signal to a 4-neuron output layer, an input vector from a
five-dimensional space is transformed or mapped to a signal vector in a four-dimensional space.

∙ Neural networks thus generate mappings from one space to another or sometimes from the input space back
to itself

28-10-2021 Neural Network : Module 1 38

28-10-2021 Neural Network : Module 1 39
•

28-10-2021 Neural Network : Module 1 40

Feedforward vs Feedback: Static vs
Dynamic
•

28-10-2021 Neural Network : Module 1 41

∙ On the other hand, recurrent or feedback networks are examples of non-linear dynamical systems.
∙ When a new input pattern is presented, the neuron outputs are computed as usual, but because these outputs
are fed back as inputs to the system, the activations of neurons get subsequently modified, leading the
network to a new state.
∙ The next state of the network is thus a function of the current input and the present state of the network, very
much like a finite state machine.

∙ In recurrent neural networks, network activations and signals are in a flux of change until (and unless) they
settle down to a steady state.

28-10-2021 Neural Network : Module 1 42

CONVEX SETS, CONVEX HULLS AND
LINEAR SEPERABILITY
•

28-10-2021 Neural Network : Module 1 43

Convex Hull

28-10-2021 Neural Network : Module 1 44

Linearly Separable
•

28-10-2021 Neural Network : Module 1 45

NON-LINEARLY SEPARABLE
PROBLEMS
•

28-10-2021 Neural Network : Module 1 46

Linearly and non linearly separable

28-10-2021 Neural Network : Module 1 47

XOR is not linearly separable

28-10-2021 Neural Network : Module 1 48

•

28-10-2021 Neural Network : Module 1 49

•

28-10-2021 Neural Network : Module 1 50

Solving the XOR Problem

28-10-2021 Neural Network : Module 1 51

MULTILAYER NETWORKS

• The threshold logic neurons can be connected into multiple layers in a feedforward fashion.

• This dramatically increases the computational capability of the system as a whole.

• In general, it is not uncommon to have more than one hidden layer when solving a classification
problem.

• Such a situation arises when one has a number of disjoint regions in the pattern space that are
associated into a single class

28-10-2021 Neural Network : Module 1 52

 The following observations are made for binary threshold neurons with two hidden layers apart from the input
and the output layer.

1. Each neuron in the first hidden layer forms a hyperplane in the input pattern space.
2. A neuron in the second hidden layer can form a hyper-region from the outputs of the first layer neurons by
performing an AND operation on the hyperplanes. These neurons can thus approximate the boundaries
between pattern classes.
3. The output layer neurons can then combine disjoint pattern classes into decision regions made by the
neurons in the second hidden layer by performing logical OR operations.

28-10-2021 Neural Network : Module 1 53

28-10-2021 Neural Network : Module 1 54
Theorem

No more than three layers in binary threshold feedforward networks are required to form
arbitrarily complex decision regions.

28-10-2021 Neural Network : Module 1 55

Proof: By Construction

28-10-2021 Neural Network : Module 1 56

Learning Algorithms
• Learning algorithms define an architecture-dependent procedure to encode pattern information into weights to
generate these internal models. Learning proceeds by modifying connection strengths.
• Most learning is data driven.
• The data might take the form of a set of input-output pattens derived from a (possibly unknown) probability
distribution.
• Here the output pattern might specify a desired system response for a given input pattern, and the issue of
learning would involve approximating the unknown function as described by the given data.
• Alternatively, the data might comprise patterns that naturally cluster into some number of unknown classes
and the learning problem might involve generating a suitable classification of the samples.
• The nature of the problem of learning allows us to demarcate learning algorithm into two categories:
supervised and unsupervised.

28-10-2021 Neural Network : Module 1 57

Supervised Learning
•

28-10-2021 Neural Network : Module 1 58

Unsupervised Learning

28-10-2021 Neural Network : Module 1 59

28-10-2021 Neural Network : Module 1 60
28-10-2021 Neural Network : Module 1 61
Error Correction and Gradient Descent
Rules
• Two kinds of supervised learning laws have evolved in the literature that employ such a philosophy:

1. Error correction rules that alter the weights of a network using a linear error measure to reduce the error in
the output generated in response to the present input pattern.
2. Gradient rules that alter the weights of a network during each pattern presentation by employing gradient
information with the objective of reducing the mean squared error (usually averaged over all training
patterns).

28-10-2021 Neural Network : Module 1 62

Learning Objective for TLNS

• A TLN is actually a linear neuron whose output is directed into a unit step or signum function.
• The neuron is adaptive when its weights are allowed to change in accordance with a well-defined learning
law.
• Commonly used adaptive algorithms for such threshold logic neurons are the Perceptron learning algorithm
and the least-mean-square (LMS) algorithm.

28-10-2021 Neural Network : Module 1 63

• The neuron input and weight vectors are assumed to be in augmented form to accommodate the neuron bias as
an adjustable weight:

28-10-2021 Neural Network : Module 1 64

•

28-10-2021 Neural Network : Module 1 65

Perceptron Learning Algorithm

28-10-2021 Neural Network : Module 1 66

A simplification with pattern set adjustment
•

28-10-2021 Neural Network : Module 1 67

•

28-10-2021 Neural Network : Module 1 68

28-10-2021 Neural Network : Module 1 69
Perceptron Convergence Theorem

28-10-2021 Neural Network : Module 1 70

28-10-2021 Neural Network : Module 1 71
28-10-2021 Neural Network : Module 1 72
28-10-2021 Neural Network : Module 1 73
28-10-2021 Neural Network : Module 1 74
28-10-2021 Neural Network : Module 1 75
 Question Bank
1. List any 4 neuronal signal function used in ANN.
2. Explain learning algorithms in neural network.
3. State and explain XOR is non-linearly separable. Also explain the implementation of XOR
function using two layered network architecture.
4. Explain learning objective for threshold logic neuron.
5. Define convex sets, convex hulls and linear separability with relevant diagrams.
6. Give any 4 differences between supervised and unsupervised learning.
7. State and prove perceptron convergence theorem.
8. What is Neural Learning? Draw and explain the general neuron model.
9. State and explain the Ex-OR problem? Also, explain how to overcome it.
10. Draw and explain architectural graph of a multi-layer perceptron with two hidden layers.

28-10-2021 Neural Network : Module 1 76