8th Term

Neural Networks
By
Engr. Muhammad Imran

1
Books:
1. Haykin, S., Neural Networks – A
Comprehensive Foundation, Second
edition or latest, McMillan.

2. Hagan, M.T., Demuth, H.B., and Beale, M.,

Neural Network Design, PWS Publishing
Company.

2
 Artificial Neural Networks (ANNs) learn by
experience rather than by modeling or
programming.
 ANN architectures are distributed, inherently
parallel and potentially real time.
 They have the ability to generalize.
 They do not require a prior understanding of the
process or phenomenon being studied.
 They can form arbitrary continuous non-linear
mappings.
 They are robust to noisy data.
 VLSI implementation is easy

3
 Pattern recognition including character
recognition, speech recognition, face recognition,
on-line signature recognition color recognition
etc.
 Weather forecasting, load forecasting.
 Intelligent routers, intelligent traffic monitoring,
intelligent filter design.
 Intelligent controller design
 Intelligent modeling.
and many more

4
 Conventional neural networks are black box
models.
 Tools for analysis and model validation are not
well established.
 An intelligent machine can only solve some
specific problem for which it is trained.
 Human brain is very complex and cannot be fully
simulated with present computing power. An
artificial neural network does not have capability
of human brain.
 Issue:
 What is the difference between human brain
neurons and other animal brain neurons?

5
A universal definition of ANNs is not available,
however, the following definition summarizes
the basic features of an ANN:
Artificial Neural Networks, also called
Neurocomputing or Parallel Distributed
Processes (PDP) or connectionist networks or
simply neural networks are interconnected
assemblies of simple processing elements, called
neurons, units or nodes, whose functionality is
loosely based on the biological neuron. The
processing ability of the network is stored in the
inter-unit connection strength, or weights,
obtained by a process of adaptation to, or
learning from a set of training patterns.

6
 A simplified view of a biological (real) neuron

7
Examples of some real neurons

8
Image of the vertical organization of neurons in the primary
visual cortex 9
The central part of a neuron is called the
soma or cell body which contains the
nucleus and the protein synthesis
machinery. The size of soma of a typical
neuron is about 10 to 80µm. Almost all the
logical functions are realized in this part of
a neuron.

10
The dendrites represent a highly branching tree
of fibers and are attached to the soma. The
word dendrite has been taken from the Greek
word dendro which means tree. Dendrites
connect the neuron to a set of other neurons.
Dendrites either receive inputs from other
neurons or connect other dendrites to the
synaptic outputs.

11
It is a long tubular fiber which divides itself
into a number of branches towards its end. Its
length can be from 100 µm to 1 m. The
function of an axon is to transmit the
generated neural activity to other neurons or
to muscle fibers. In other words, it is output
channel of the neuron. The point where the
axon is connected to its cell body is called the
Hillock zone.

12
The junction at which a signal is passed from one
neuron to the next is called a synapse (from the
Greek verb to join). It has a button like shape
with diameter around 1 µm. Usually a synapse
is not a physical connection (the axon and the
dendrite do not touch) but there is a gap called
the synaptic cleft that is normally between
200Å to 500Å (1Å = 10-10 m). The strength of
synaptic connection between neurons can be
chemically altered by the brain in response to
favorable and unfavorable stimuli in such a
way as to adapt the organism to function
optimally within its environment.

13
“The nerve fibre is clearly a signalling mechanism of limited scope.
It can only transmit a succession of brief explosive waves, and the
message can only be varied by changes in the frequency and in the
total number of these waves. … But this limitation is really a small
matter, for in the body the nervous units do not act in isolation as
they do in our experiments. A sensory stimulus will usually affect a
number of receptor organs, and its result will depend on the
composite message in many nerve fibres.” Lord Adrian, Nobel
Acceptance Speech, 1932.
14
 Single neurons are highly complex
electrochemical devices
 Synaptically connected networks are only part
of the story
 Many forms of interneuron communication
now known – acting over many different
spatial and temporal scales

15
 Single neuron activity

• Membrane potential is the voltage difference between a neuron

and its surroundings (0 mV)

Membrane potential

Cell Cell

Cell Cell

0 Mv

16
 Single neuron activity

•If you measure the membrane potential of a neuron and print it out
on the screen, it looks like:

spike

17
 Single neuron activity

•A spike is generated when the membrane potential is greater than

its threshold

18
 Abstraction
•So we can forget all sub-threshold activity and concentrate on
spikes (action potentials), which are the signals sent to other
neurons

Spikes

19
• Only spikes are important since other neurons receive them
(signals)

• Neurons communicate with spikes

• Information is coded by spikes

• So if we can manage to measure the spiking

time, we decipher how the brain works ….

20
 Again its not quite
that simple

• spiking time in the cortex is random

21
With identical input
for the identical neuron

spike patterns are similar, but not identical

22
Recording from a real neuron: membrane potential

23
 Single spiking time is meaningless
To extract useful information, we have to average
 for a group of neurons in a local circuit where neuron
codes the same information
 over a time window

to obtain the firing rate r

r =
=
Local circuit

= 6 Hz Time window = 1 sec

24
Hence we have firing rate of a group of neurons

So we can have a network of these

r1 local groups

w1: synaptic strength

R = f (∑ w j r j )

wn
rn

25
 ri is the firing rate of input local circuit

The neurons at output local circuits receive signals in the form

N

∑wr
i=1
i i

The output firing rate of the output local circuit is then given by
R N
R = f (∑wi ri )
i =1

where f is the activation function, generally a Sigmoidal

function of some sort
wi weight, (synaptic strength) measuring the strength of the
interaction between neurons.
26
Artificial Neural networks

Local circuits (average to get firing rates)

Single neuron (send out spikes)

27
No, with present computing power is it almost
impossible to simulate a full behavior of
human brain.
Reasons:
 It is estimated that the human cerebral
cortex contains 100 billion neurons. Each
neuron has as many as 1000 dendrites and,
hence within the cerebral cortex there are
approximately 100,000 billion synapses.
 The behavior of the real nervous system is
very complex and not yet fully known.

28
x1 w1

Activation
function
y
x2 w2 ∑ f

xp wp  p 
 ∑w i p i + b 
y =f 
 i =1 
29
For simplicity, the threshold contribution b may be treated
as an extra input to the neuron, as shown below, where x0 = 1
w0 = b.
x0 w0

x1 w1 Activation
function
y
x2 w2 ∑ f

In this case

p 
xp wp y = f ∑ ( w i xi ) 
 i=0  30
The activation function of a neuron, also
known as transfer function, may be linear or
non-linear. A particular activation function
is chosen to satisfy some specification of the
problem that the neuron is attempting to
solve.
A variety of activation functions have been
proposed. Some of these are discussed
below:

31
(a) The hard limit activation function: This
activation function sets the output of the
neuron to 0 if the function argument is less
than 0, or 1 if its argument is greater than or
equal to 0. (see figure below). This function
is generally used to create neurons that
classify inputs into two distinct classes.

32
Example 1: The input to a neuron is 3, and its
weight is 1.8.
(a) What is the net input to the activation
function?
(b) What is the output of the neuron if it has the
hard limit transfer function?
Solution:
(a) Net input = u = 1.8×3 = 5.4

(b) Neuron output = f(u) = 1.

33
Example 2: Repeat example 1 if its bias is
(i) –2 (ii) –6
Solution: (a) net input = u = (1.8×3) + (-2)
= 5.4 – 2 = 3.4
output = f(u) = 1.0
(b) net input = u = (1.8×3) + (-6)
= 5.4 – 6 = -0.6
output = 0.

34
Example 3: Given a two input neuron with
the following parameters: b = 1.4, W = [1.2
3.4] and P = [-3 5], calculate the neuron
output for the hard limit transfer function.
Solution: net input to the activation function
=
u =[ ]

−3
u =[1.2output
Neuron 3.4]=f(u)=+1.4 =14.8
1.0
5 

35
 (b) Symmetrical hard limit:
This activation function is defined as follows:
y = -1 u < 0
y = +1 u ≥ 0
Where u is the net input and y is the output of
the function (see figure below):

1
y

-1

0 u
36
Example 4: Given a two output neuron with
the following parameters: b = 1.2, W = [3
2], and p = [-5 6]. Calculate the neuron
output for the symmetrical hard limit
activation function.
Solution:
− 5
u = [3 2]  +1.2 = −1.8
6 

Output = y = f(u) = -1

37
(c) Linear activation function:
The output of a linear activation function is
equal to its input:
y=u
as shown in the following figure:

0 u

38
(d) The log-sigmoid activation function:
This activation function takes the input
(which may have any value between plus
and minus infinity) and squashes the output
into the range 0 to 1, according to the
expression:

1 A plot of this function is given

y= −u
below:
1 +e
1

0 39
0 u
Example 5: Repeat example 2 for a log-
sigmoid activation function.
Solution:
(a) net input = u = 3.4
output = f(u) = 1/(1 + e-3.4) = 0.9677
(b) net input u = -0.6
output = f(u) = 1/(1 + e0.6) = 0.3543
Example 6: Repeat example 3 for a log
sigmoid activation function.
Solution: output = 1

40
(e) Hyperbolic Tangent Sigmoid:
This activation function is shown in the
following figure and is defined
mathematically as follows:

eu − e − u
y = u −u
e +e
1

-1 0 41
 +1
+1
0
-1
-1
Gaussian function Schmitt trigger Saturation Limiter

and many more

42
RULES FOR SIGNAL FLOW
GRAPH
 Rule 1: A signal flows along a link
only in the direction defined by
the arrow on the link. (see fig a:
synaptic link, fig b: activation
link)
 Rule 2: A node signal equals the
algebraic sum of all signals
entering the pertinent node via
the incoming links. (see fig c)
 Rule 3: The signal at a node is
transmitted to each outgoing link
originating from the node, with
the transmission being entirely
independent of the transfer
function of the outgoing links.
(see fig d)

43
 The block diagram of a
neuron shown in figure,
provides a functional
description of the
various elements that
constitute the model of
an artificial neuron.
 We can simplify the
appearance by using
signal flow graph.

44
 Feedback is said to exist in a dynamic system
whenever the output of an element in the
system influences in the input applied to that
particular element, thereby giving rise to one or
more closed paths for the transmission of
signals around the system.
 The Open loop architectural graph of a neuron

45
46
Connecting several neurons in some specific
manner yields an artificial neural network.
 The architecture of an ANN defines the
network structure, that is the number of
neurons in the network and their
interconnectivity.
 In a typical ANN architecture, the artificial
neurons are connected in layers and they
operate in parallel.
 The weights or the strength of connection
between the neurons are adapted during
use to yield good performance.
 Each ANN architecture has its own
learning rule.

47
(a) Single Layer feed forward networks:

Input layer Output layer

48
(b) Multilayer Feed forward Networks

Fully connected feed forward network

with one hidden layer

Input layer Hidden layer Output layer 49

Partially connected feedforward network
(b) Multilayer Feedforward Networks
with one hidden layer

Input
signal

Input layer Hidden layer Output layer 50

Recurrent network with no hidden neurons

51
52
53