CSC445: Neural Networks

CSC445: Neural Networks
CHAPTER 00
INTRODUCTION
Prof. Dr. Mostafa Gadal-Haqq M. Mostafa

Computer Science Department
Faculty of Computer & Information Sciences
AIN SHAMS UNIVERSITY
(All figures in this presentation are copyrighted to Pearson Education, Inc.)

Course Objectives
 Introduce the main concepts and techniques of
neural network systems.
 Investigate the principal neural network
models and applications.
ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 2

Textbooks
 Recommended
 S. Haykin. Neural Networks and Learning Machines, 3ed.,
Prentice Hall (Pearson), 2009.
 Comprehensive and up-to-date
 L. Fausett. Fundamental of Neural Networks: Architectures,

Algorithms, and Applications. Prentice Hall, 1995.
 Intermediate

Course Assessment
 Assessment
 Homework, Quizzes, Computer Assignments (10 points)
 Midterm Exam (15 Points)
 Project + Lab Exam (10 Points)
 Final Exam (65 Points)
 Programming and homework assignments

 Late answers are NOT accepted!

Good Practice
 How to pass this course?
 Don’t accumulate …
 Do it yourself …
 Ask for help …

Resources
 Webpages:
 The Neural Networks FAQs

 ftp://ftp.sas.com/pub/neural/FAQ.html
 The Neural Networks Resources

 http://www.neoxi.com/NNR/index.html

Introduction
 What is a Neural Network?
 Benefits of Neural Networks !
 The Human Brain
 Models of A Neuron
 Neural Networks and Graphs
 Feedback
 Network Architectures
 Knowledge Representation
 Learning Processes
 Learning Tasks

What is a Neural Network?
 A neural network is a massively parallel distributed
processor made up of simple processing units
(neurons) that has a natural tendency for storing
experiential knowledge and making it available for
use.
 It resembles the brain in two respects:

 Knowledge is acquired by the network from its environment
through the learning process.
 Interneuron connection strengths, known as synaptic weights,
are used to store the acquired knowledge.

Benefits of Neural Networks
 Nonlinearity (NN could be linear or nonlinear)
 A highly important property, particularly if the underlying
physical mechanism responsible for generation of the input
signal (e.g. speech signal) is inherently nonlinear.
 Input-Output Mapping
 It finds the optimal mapping between an input signal and the
output results through learning mechanism that adjust the
weights that minimize the difference between the actual
response and the desired one. (nonparametric statistical
inference).
 Adaptivity
 A neural network could be designed to change its weights in real
time, which enables the system to operate in a nonstationary
environment.
 Evidential Response
 In the context of pattern classification, a network can be
designed to provide information not only about which
particular pattern to select, but also about the confidence in the
decision made.
 Contextual Information
 Every neuron in the network is potentially affected by the global
activity of all other neurons. Consequently, contextual
information is dealt with naturally by a neural network.
 Fault tolerance
 A neural network is inherently fault tolerant, or capable of
robust computation, in the sense that its performance degrades
gracefully under adverse operating conditions.

 VLSI Implementation
 The massively parallel nature of a neural network, makes it
potentially fast for a certain computation task, which also makes
it well suited for implementation using VLSI technology.
 Uniformity of Analysis and Design
 Neural networks enjoy universality as information processors:
 Neurons, in one form or another, represent an ingredient common
to all neural networks, which makes it possible to share theories and
learning algorithms in different applications.
 Modular networks can be built through a seamless integration of
modules
 Neurobiology Analogy
 The design of a neural network is motivated by analogy to the
brain (the fastest and powerful fault tolerant parallel processor).
Application of Neural Networks
 Neural networks are tractable and easy to
implement, Specially in hardware. This made it

attractive to be used in a wide range of applications:
 Pattern Classifications
 Medical Applications
 Forecasting
 Adaptive Filtering
 Adaptive Control

The Human Nervous System
 The human nervous system may be viewed as a
three-stage system:
 The brain, represented by the neural net, is central to the
system. It continually receive information, perceives it, and
makes appropriate decision.
 The receptors convert stimuli from the human body or the
external environment into electrical impulses that convey
information to the brain.
 The effectors convert electrical impulses generated by the brain
into distinct responses as system output
Figure 1 Block diagram representation of nervous system.

the Human Brain
 There are approximately 10 billion (1010) neurons in the
human cortex, compared with 10 of thousands (104)of
processors in the most powerful parallel computers.
 Each biological neuron is connected to several thousands of
other neurons.
 The typical operating speeds of biological neurons is
measured in milliseconds (10-3 s), while a silicon chip can
operate in nanoseconds (10-9 s). (Lack of processing units in
computers can be compensated by speed.)
 The human brain is extremely energy efficient, using
approximately 10-16 joules per operation per second, whereas
the best computers today use around 10-6 joules per
operation per second.
The Natural Neural Cell
 The neuron’s cell body (soma)
processes the incoming activations
and converts them into output
activations.
 Dendrites are fibers which emanate
from the cell body and provide the
receptive zones that receive
activation from other neurons.
 Axons are fibers acting as
transmission lines that send
activation to other neurons.
 The junctions that allow signal
transmission between the axons and
dendrites are called synapses. The
process of transmission is by
diffusion of chemicals called
neurotransmitters across the
synaptic cleft Figure 2 The pyramidal cell.
Organization of the Human Brain
 Neural Microcircuit
 An assembly of synapses organized into patterns of
connectivity to produce a functional operation of
interest.
 Local circuits
 Group of neurons with similar or different properties
that perform operations characteristic of a localized
region in the brain.
 Interregional circuits
 Pathways, columns, and topographic maps, which
involve multiple regions located in different parts of the
brain.
 Topographic maps are organized to respond to
incoming sensory information
Figure 3 Structural organization of levels in the brain.

Modularity of the Human Brain
Figure 4 Cytoarchitectural map of the cerebral cortex. The different areas are identified by
the thickness of their layers and types of cells within them. Some of the key sensory areas
are as follows: Motor cortex: motor strip, area 4; premotor area, area 6; frontal eye fields,
area 8. Somatosensory cortex: areas 3, 1, and 2. Visual cortex: areas 17, 18, and 19.
Auditory cortex: areas 41 and 42. (From A. Brodal, 1981; with permission of Oxford
University Press.)

Brief History of ANN
1943 McCulloch and Pitts proposed the McCulloch-Pitts neuron model
1949 Hebb published his book The Organization of Behavior, in which the Hebbian
learning rule was proposed.
1958 Rosenblatt introduced the simple single layer networks now called Perceptrons.
1969 Minsky and Papert’s book Perceptrons demonstrated the limitation of single
layer perceptrons, and almost the whole field went into hibernation.
1982 Hopfield published a series of papers on Hopfield networks.
1982 Kohonen developed the Self-Organising Maps that now bear his name.
1986 The Back-Propagation learning algorithm for Multi-Layer Perceptrons was

rediscovered and the whole field took off again.
1990s The sub-field of Radial Basis Function Networks was developed.
2000s The power of Ensembles of Neural Networks and Support Vector Machines
becomes apparent.

Models of a Neuron
 The artificial neuron is
made up of three basic
elements:
 A set of synapses, or
connecting links, each of
which is characterized by a
weight or strength of its own
 An adder for summing the
input signals, weighted by the
respective synaptic weight.
Figure 5 Nonlinear model of a
 An activation function for
neuron, labeled k.
limiting the amplitude of the
output of a neuron. (Also
called squashing function)
Models of a Neuron
 In mathematical terms:
 The output of the summing
function is the linear
combiner output:
m
uk   wkj x j
j 1
 and the final output signal of
the neuron:
yk   (uk  bk ) Figure 5 Nonlinear model of a
neuron, labeled k.
 (.) is the activation

function.

Effect of a Bias
 The use of a bias bk has the
effect of applying affine
transformation to the output uk.
vk  uk  bk
 That is, the bias value changes
the relation between the induced
local field, or the activation
potential vk, and the linear
combiner uk as shown in Fig. 6.
Figure 6 Affine transformation
produced by the presence of a
bias; note that vk = bk at uk = 0.

Models of a Neuron
 Then we may write:
m
vk   wkj x j
j 0
and
yk   (vk )
where
x0  1, and wk 0  bk
Figure 7 Another nonlinear model of a
neuron; wk0 accounts for the bias bk.

Types of Activation Function
 Threshold function
also called Heaviside
function.
1 if v  0
 (v )  
0 if v  0
 This model is the
McCulloch and Pitts Figure 8 (a) Threshold function.
neuron model.
 That is, the neuron will has output signal only if its
activation potential is non-negative, a property
known as all-or-none.

 Sigmoid function
 the most commonly used function. It is a strictly increasing
function that exhibit a graceful balance between linear and
nonlinear behavior.
1
 (v ) 
1  eav
 a is the slope parameter.
Figure 8(b) Sigmoid function for
 This function is differentiable, varying slope parameter a.
which is an important feature for the neural network theory.

 Other activation functions
 the signum function, which is an odd function of its
activation potential v.
 1 if v  0

 (v)   0 if v  0
 1 if v  0

 The hyperbolic tangent function , which allows for an
odd sigmoid-type function.
 (v)  tanh(v)

Stochastic Model of a Neuron
 The previous neuron models are deterministic, that
is, its output is precisely known for any input signal.
 In some applications it is desirable to have a
stochastic neuron model.
 That is, the neuron is permitted to reside in only one
of two states; +1 and -1, say. The decision for a

neuron to fire (i.e., switch its state from “off” to
“on”) is probabilistic.

Stochastic Model of a Neuron
 Let x denote the state of the neuron, and P(v)
denotes the probability of firing, where v is the
activation potential, then we may write:
 1 with probabilit y P(v )
x
 1 with probabilit y 1-P(v )
 A standard choice of P(v)is the sigmoid function

1
P (v )  v / T
1 e
 Where T is a parameter used to control the noise
level and therefore the uncertainty in firing. When
T 0, the model reduces to the deterministic model.
Neural Networks Viewed as Directed Graph
 Figures 5 and 7 are functional description
of a neuron. We can use signal-flow graph
(a network of directed links that are
interconnected to points called nodes) to
describe the network using these rules:
 A signal flow along a link only in the direction defined
by the arrow on the link
 A node signal equal the algebraic sum of all signals
entering the node.
 The signal at a node is transmitted to each outgoing
link originated from that node.
Figure 9 illustrating basic rules for the construction of signal-flow graphs.

 A Signal-Flow graph of a neuron:
Figure 10 Signal-flow graph of a neuron.

 An Architectural graph of a neuron:
Figure 11 Architectural graph of a neuron.

 We have three graphical representations of a neural
network.
 Block diagram (Functional)
 Complete directed graph (Signal-Flow)
 Partially Complete directed graph (Architectural)
Functional Signal-flow Architectural

Feedback
 Feedback is said to exist in dynamic systems
whenever the output of a node influences in part the
input of that particular node. (Very important in the
study of recurrent networks.
yk (n)  A[ xj (n)]

and
xj (n)  x j (n)  B[ yk (n)]
Figure 12 Signal-flow graph of a single-
A loop feedback system.
yk (n)  [ x j (n)]
1  AB
Network Architectures
 Single-layer Feedforward
Networks
 Note that: we do not count the input
layer of the source nodes because no
computation is performed there.
Figure 15 Feedforward network

with a single layer of neurons.

 Multilayer Feedforward Networks
 Input layer (source nodes)
 One or more hidden layers
 One output layer
 It is referred to as 10-4-2 network.
 Could be fully or partially connected.
 By adding one or more hidden

layers, the network is enabled to
extract higher order statistics from
its input).
Figure 16 Fully connected feedforward

network with one hidden layer and one
output layer.
 Recurrent Networks
 A recurrent neural networks
should has at least one
feedback loop
 Self-feedback refers to a
situation where the output
of a node is feedback into its
own input.
Figure 17 Recurrent network with no self-

feedback loops and no hidden neurons.
 Recurrent Networks
 A recurrent neural
networks with self
feedback loop and with
hidden neurons.
Figure 18 Recurrent network with hidden neurons.

Knowledge Representation
 Knowledge refers to
 stored information or models used by a person or

machine to interpret, predict, and approximately respond
to the outside world.
 Characteristics of knowledge representation:
 What information is actually made explicit?
 How the information is physically encoded for subsequent use?

 The major task for a neural network is
 to learn a model of the world (environment) in which it is

embedded, and to maintain the model sufficiently consistently
with the real world so as to achieve the specific goals of the
application of interest.
 knowledge of real world consists of:
 the known world state represented by facts (prior information).
 Observations of the world obtained by sensors (measurements).

These observations are used to train the neural network.

 (Commonsense) Roles of Knowledge Representation
 Role 1: Similar inputs form similar
classes should usually produce
similar representation.
 Role 2: Items to be categorized as
separate classes should be given
widely different representation.
 Role 3: if a particular feature is
important, then there should be a
large number of neurons are
involved in the representation. Figure 19 Illustrating the
 Role 4: Prior information and relationship between inner product
invariances should be built into the and Euclidean distance as
design of a NN when they are measures of similarity between
patterns.
available.

 How to Build prior Info into Neural Network Design?
 Currently ,there are no well-defined
rules. But rather some ad hoc
procedures:
 Restricting the network
architecture, which is achieved
through the use of local
connections (receptive fields).
 Constraining the choice of
synaptic weights, which is
implemented through the use of
weight-sharing.
 NNs that make use of these two
techniques are called Convolutional
Networks.
Figure 20: Illustrating the combined use of a receptive field and weight sharing. All four hidden
neurons share the same set of weights exactly for their six synaptic connections.
 How to Build Invariance into Neural Network Design?
 Invariance by Structure
 For example, enforcing in-plain rotation by making wij = wji
 Invariance by Training
 Including different examples of each class.
 Invariant Feature Space
 Transforming the data to invariant feature space (Fig. 21).
Figure 21 Block diagram of an invariant-feature-space type of system.

Neural Networks vs. Pattern Classifiers
 Pattern Classifiers: We should have a prior
knowledge about the data to be able to explicitly
model it.
(Explicit)
Data Model Training
Building
 Neural Networks: The data set speaks about itself.

The NN learns the implicit model in the dat.
Data Training

Learning Processes
 Supervised Learning: Learning with a Teacher
Figure 24 Block diagram of learning with a teacher; the part of the

figure printed in red constitutes a feedback loop.

Learning Processes
 Unsupervised Learning: Learning without a Teacher
Figure 26 Block diagram of unsupervised learning.

Learning Processes
 Reinforcement Learning: Learning without a Teacher
Figure 25 Block diagram of reinforcement learning; the learning

system and the environment are both inside the feedback loop.

Learning Tasks
 Pattern Association
Figure 27 Input–output relation of pattern

associator.

Learning Tasks
 Pattern Recognition
Figure 28 Illustration of the classical approach to pattern classification.

Learning Tasks
 Function Approximation: System Identification
Figure 29 Block diagram of system identification:

The neural network, doing the identification, is
part of the feedback loop.

Learning Tasks
 Function Approximation: Inverse Modeling
Figure 30 Block diagram of inverse system modeling. The neural network,

acting as the inverse model, is part of the feedback loop.

Learning Tasks
 Control
Figure 31 Block diagram of feedback control system.

Next Time
Rosenblatt’s Perceptron
51

CSC445: Neural Networks

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

CSC445: Neural Networks

Uploaded by

Copyright:

Available Formats

CSC445: Neural Networks

Prof. Dr. Mostafa Gadal-Haqq M. Mostafa

(All figures in this presentation are copyrighted to Pearson Education, Inc.)

 Introduce the main concepts and techniques of

neural network systems.

 Investigate the principal neural network

models and applications.

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 2

 L. Fausett. Fundamental of Neural Networks: Architectures,

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 3

 Homework, Quizzes, Computer Assignments (10 points)

 Midterm Exam (15 Points)

 Project + Lab Exam (10 Points)

 Final Exam (65 Points)

 Programming and homework assignments

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 4

 How to pass this course?

 Ask for help …

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 5

 The Neural Networks FAQs

 The Neural Networks Resources

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 6

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 7

 It resembles the brain in two respects:

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 8

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 10

implement, Specially in hardware. This made it

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 12

Figure 1 Block diagram representation of nervous system.

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 13

Figure 3 Structural organization of levels in the brain.

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 17

1982 Hopfield published a series of papers on Hopfield networks.

1986 The Back-Propagation learning algorithm for Multi-Layer Perceptrons was

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 18

 (.) is the activation

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 20

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 21

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 22

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 23

which is an important feature for the neural network theory.

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 24

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 25

is, its output is precisely known for any input signal.

 In some applications it is desirable to have a

stochastic neuron model.

 That is, the neuron is permitted to reside in only one

of two states; +1 and -1, say. The decision for a

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 26

 A standard choice of P(v)is the sigmoid function

Figure 9 illustrating basic rules for the construction of signal-flow graphs.

Figure 10 Signal-flow graph of a neuron.

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 29

Figure 11 Architectural graph of a neuron.

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 30

Functional Signal-flow Architectural

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 31

yk (n)  A[ xj (n)]

Figure 15 Feedforward network

ASU-CSC445: Neural Networks Prof. Dr. Mostafa Gadal-Haqq 33

 By adding one or more hidden

Figure 16 Fully connected feedforward

Figure 17 Recurrent network with no self-