Artificial Neural Networks

Introduction
 My whereabouts

• Dr. M. Castellani
• Centria – Centre for Artificial Intelligence
• Room 2.50
• Ext. 10757
• Email: mcas@fct.unl.pt
• Research Interests
– Machine Learning, Approximate Reasoning
– Evolutionary techniques for generation of Fuzzy Logic Systems
– Evolutionary techniques for generation of Neural Networks Systems
– Spiking Neural Network models.
– Data Classification, Time Series Modelling and Control.

30/09/22 Artificial Neural Networks - I 2

 Table of Contents

• Introduction to ANNs
– Taxonomy
– Features
– Learning
– Applications
I

• Supervised ANNs • Unsupervised ANNs

– Examples – Examples
– Applications – Applications
– Further topics – Further topics
II III

30/09/22 Artificial Neural Networks - I 3

 Contents - I

• Introduction to ANNs
– Processing elements (neurons)
– Architecture
• Functional Taxonomy of ANNs
• Structural Taxonomy of ANNs
• Features
• Learning Paradigms
• Applications

30/09/22 Artificial Neural Networks - I 4

 The Biological Neuron

• 10 billion neurons in human brain • 106 billion synapses in human brain

• Summation of input stimuli • Chemical transmission and
– Spatial (signals) modulation of signals
– Temporal (pulses) • Inhibitory synapses
• Threshold over composed inputs • Excitatory synapses
• Constant firing strength

30/09/22 Artificial Neural Networks - I 5

 Biological Neural Networks

• 100˙000 synapses per

neuron
• Computational power =
connectivity
• Plasticity
– new connections
– strength of connections
modified

30/09/22 Artificial Neural Networks - I 6

 Neural Dynamics

40 mV membrane
rest
20
activation

0
Action potential
-20
Action potential ≈ 100mV
-40
Activation threshold ≈ 20-30mV
-60 Rest potential ≈ -65mV
Spike time ≈ 1-2ms
-80 Refractory time ≈ 10-20ms
Refractory time
-100
ms
-120
0 10 20 30 40 50 60 70 80 90 100

30/09/22 Artificial Neural Networks - I 7

 The Artificial Neuron

x1(t)
Stimulus wi1
ui t    wij  x j t  x2(t)
j wi2
wi3
x3(t) w ij  x j (t ) yi  f (ui (t)) yi(t)
wi4
j

Response
x4(t) wi5
yi t   f urest  ui t 
x5(t)
Neuron i

urest = resting potential

xj(t) = output of neuron j at time t
wij = connection strength between neuron i and neuron j
u(t) = total stimulus at time t

30/09/22 Artificial Neural Networks - I 8

 Artificial Neural Models

• McCulloch Pitt-type Neurons (static)

– Digital neurons: activation state interpretation
(snapshot of the system each time a unit fires)
– Analog neurons: firing rate interpretation
(activation of units equal to firing rate)
– Activation of neurons encodes information
• Spiking Neurons (dynamic)
– Firing pattern interpretation (spike trains of
units)
– Timing of spike trains encodes information
(time to first spike, phase of signal, correlation
and synchronicity

30/09/22 Artificial Neural Networks - I 9

 Binary Neurons
hard threshold

1.2
output Stimulus Response
yi  f urest  ui 
1
0.8
on ui   wij  x j
0.6
j
0.4
0.2
input
0
-0.2 -10
“Hard” threshold
-8 -6 -4 -2 0 2 4 6 8 10

-0.4
heaviside
-0.6
 z    ON 
-0.8
 
-1
off
f z    
-1.2  else  OFF  = threshold
 

• ex: Perceptrons, Hopfield NNs, Boltzmann

Machines
• Main drawbacks: can only map binary
functions, biologically implausible.
30/09/22 Artificial Neural Networks - I 10
 Analog Neurons
sigmoid

output
1.2
Stimulus Response
1 on
0.8
ui   wij  x j yi  f urest  ui 
0.6
j
0.4

0.2
input
0
-10 -8 -6 -4 -2 0 2 4 6 8 10
-0.2

-0.4
2/(1+exp(-x))-1 “Soft” threshold
-0.6
-0.8
2
-1
off
f z   1
-1.2 1  e z

• ex: MLPs, Recurrent NNs, RBF NNs...

• Main drawbacks: difficult to process time
patterns, biologically implausible.

30/09/22 Artificial Neural Networks - I 11

 Spiking Neurons

Stimulus  = spike and afterspike potential

urest = resting potential
ui t    wij  x j t  (t,u() = trace at time t of input at time 
j = threshold
xj(t) = output of neuron j at time t
wij = efficacy of synapse from neuron i to
neuron j
Response u(t) = input stimulus at time t

 
yi (t )  f urest   (t  t f )    0  t , ui  
t

 dz 
 z   &  0  ON 
f z    dt

 
 else  OFF 

30/09/22 Artificial Neural Networks - I 12

 Spiking Neuron Dynamics

neuron output y(t)


2.5 V urest+(t-tf)
2

1.5

0.5 t

0
0 10 20 30 40 50 60 70 80 90 100
-0.5

-1

30/09/22 Artificial Neural Networks - I 13

 Artificial Neural Networks

Output layer

fully connected
Hidden layers

Input layer

sparsely connected

30/09/22 Artificial Neural Networks - I 14

 Feedforward ANN Architectures

• Information flow unidirectional

• Static mapping: y=f(x)
• Multi-Layer Perceptron (MLP)
• Radial Basis Function (RBF)
• Kohonen Self-Organising Map
(SOM)

30/09/22 Artificial Neural Networks - I 15

 Recurrent ANN Architectures

• Feedback connections
• Dynamic memory:
y(t+1)=f(x(τ),y(τ),s(τ)) τ(t,t-1,...)
• Jordan/Elman ANNs
• Hopfield
• Adaptive Resonance Theory (ART)

30/09/22 Artificial Neural Networks - I 16

 ANN Capabilities

• Learning
• Approximate reasoning
• Generalisation capability
• Noise filtering
• Parallel processing
• Distributed knowledge base
• Fault tolerance

30/09/22 Artificial Neural Networks - I 17

 Main Problems with ANN

• Knowledge base not transparent

(black box)
• Learning sometimes difficult/slow
• Limited storage capability

30/09/22 Artificial Neural Networks - I 18

 ANN Learning Paradigms

• Supervised learning
– Classification
– Control
– Function approximation
– Associative memory
• Unsupervised learning
– Clustering
• Reinforcement learning
– Control
30/09/22 Artificial Neural Networks - I 19
 Supervised Learning

• Teacher presents ANN input-output

pairs
• ANN weights adjusted according to
error
• Iterative algorithms (e.g. Delta rule,
BP rule)
• One-shot learning (Hopfield)
• Quality of training examples is
critical
30/09/22 Artificial Neural Networks - I 20
 Delta Rule
=learning coefficient
wij=connection from neuron xj to yi
e  d i  yi x=(x1,x2,...,xn) ANN input
y=(y1,y2,...,yn) ANN output
wij    e  x j d=(d1,d2,...,dn) desired output
(x,d) training example
e=ANN error

• a.k.a. Least Mean Squares y1 y2 y3

• Widrow-Hoff iterative delta rule
(1960)
• Gradient descent of the error surface
• Guaranteed to find minimum error w11 w12 w13 w14
configuration in single layer ANNs
• Stochastic approximation of desired x1 x2 x3 x4
behaviour
30/09/22 Artificial Neural Networks - I 21
 Unsupervised Learning

• ANN adapts weights to cluster input

data
• Hebbian learning
– Connection stimulus-response
strengthened (hebbian)
• Competitive learning algorithms
– Kohonen & ART
– Input weights adjusted to resemble
stimulus

30/09/22 Artificial Neural Networks - I 22

 Hebbian Learning
General Formulation Hebb postulate Kohonen & Grossberg (ART)

F    yi  x j F    yi  x j  wij 
d
wij  F wij , yi , x j  d d
dt wij    yi  x j wij    yi  x j  wij 
dt dt

=learning coefficient
wij=connection from neuron xj to yi y1

• Hebb postulate (1948)

• Correlation-based learning
• Connections between concurrently w11 w12
firing neurons are strengthened
• Experimentally verified (1973)
x1 x2
30/09/22 Artificial Neural Networks - I 23
 Reinforcement Learning

• Sequential tasks
• Desired action may not be known
• Critic evaluation of ANN behaviour
• Weights adjusted according to critic
• May require credit assignment
• Population-based learning
– Evolutionary Algorithms
– Swarming Techniques
– Immune Networks
30/09/22 Artificial Neural Networks - I 24
 ANN Summary

Artificial Neural Networks

Feedforward Recurrent

Unsupervised Supervised Unsupervised Supervised

(Kohonen) (MLP, RBF) (ART) (Elman, Jordan,
Hopfield)

30/09/22 Artificial Neural Networks - I 25

 ANN Application Areas

• Classification
• Clustering
• Associative memory
• Control
• Function approximation

30/09/22 Artificial Neural Networks - I 26

 ANN Classifier systems

• Learning capability
• Statistical classifier systems
• Data driven
• Generalisation capability
• Handle and filter large input data
• Reconstruct noisy and incomplete
patterns
• Classification rules not transparent
30/09/22 Artificial Neural Networks - I 27
 Applications for ANN
Classifiers
• Pattern recognition
– Industrial inspection
– Fault diagnosis
– Image recognition
– Target recognition
– Speech recognition
– Natural language processing
• Character recognition
– Handwriting recognition
– Automatic text-to-speech conversion

30/09/22 Artificial Neural Networks - I 28

 Clustering with ANNs

• Fast parallel distributed processing

• Handle large input information
• Robust to noise and incomplete
patterns
• Data driven
• Plasticity/Adaptation
• Visualisation of results
• Accuracy sometimes poor
30/09/22 Artificial Neural Networks - I 29
 ANN Clustering Applications

• Natural language processing

– Document clustering
– Document retrieval
– Automatic query
• Image segmentation
• Data mining
– Data set partitioning
– Detection of emerging clusters
• Fuzzy partitioning
• Condition-action association
30/09/22 Artificial Neural Networks - I 30
 Associative ANN Memories

• Stimulus-response association
• Auto-associative memory
• Content addressable memory
• Fast parallel distributed processing
• Robust to noise and incomplete
patterns
• Limited storage capability

30/09/22 Artificial Neural Networks - I 31

 Application of ANN Associative
Memories
• Character recognition
• Handwriting recognition
• Noise filtering
• Data compression
• Information retrieval

30/09/22 Artificial Neural Networks - I 32

 ANN Control Systems

• Learning/adaptation capability
• Data driven
• Non-linear mapping
• Fast response
• Fault tolerance
• Generalisation capability
• Handle and filter large input data
• Reconstruct noisy and incomplete patterns
• Control rules not transparent
• Learning may be problematic

30/09/22 Artificial Neural Networks - I 33

 ANN Control Schemes

• ANN controller
• conventional controller + ANN for
unknown or non-linear dynamics
• Indirect control schemes
– ANN models direct plant dynamics
– ANN models inverse plant dynamics

30/09/22 Artificial Neural Networks - I 34

 ANN Control Applications

• Non-linear process control

– Chemical reaction control
– Industrial process control
– Water treatment
– Intensive care of patients
• Servo control
– Robot manipulators
– Autonomous vehicles
– Automotive control
• Dynamic system control
– Helicopter flight control
– Underwater robot control

30/09/22 Artificial Neural Networks - I 35

 ANN Function Modelling

• ANN as universal function approximator

• Dynamic system modelling
• Learning capability
• Data driven
• Non-linear mapping
• Generalisation capability
• Handle and filter large input data
• Reconstruct noisy and incomplete inputs

30/09/22 Artificial Neural Networks - I 36

 ANN Modelling Applications

• Modelling of highly nonlinear

industrial processes
• Financial market prediction
• Weather forecasts
• River flow prediction
• Fault/breakage prediction
• Monitoring of critically ill patients

30/09/22 Artificial Neural Networks - I 37

 References

• B. Kosko (1992), Neural Networks and

Fuzzy Systems, Prentice-Hall Int. Ed.
• R. Rojas (1992), Neural Networks,
Springer Verlag
• P. Mehra and B. W. Wah 1992, Artificial
Neural Networks: Concepts And Theory,
Ieee Computer Society Press.
• M. H. Hassoun (1995), Fundamentals of
Artificial Neural Networks, MIT Press.

30/09/22 Artificial Neural Networks - I 38