Neural Networks

A Comprehensive Foundation

• Introduction
• Learning Processes
• Single Layer Perceptrons
• Multilayer Perceptrons
• Reference: Neural Networks: A Comprehensive Foundation
Simon Haykin, 2nd Edition, Prentice Hall

• mroushdy@cis.asu.edu.eg
• Mohamed.Roushdy@fue.edu.eg
• miroushdy@hotmail.com
Overview
• Neural Network (NN) or Artificial Neural Networks
(ANN) is a computing paradigm
• The key element of this paradigm is
• the novel structure of the information processing
system consisting of a large number of highly
interconnected processing elements (neurons) working
in unison to solve specific problems
• Development of NNs date back to the early 1940s
• Minsky and Papert, published a book (in 1969)
• summed up a general feeling of frustration (against
neural networks) among researchers
Overview (Contd.)
• Experienced an upsurge in popularity in the late
1980s
• Result of the discovery of new techniques and
developments and general advances in computer
hardware technology
• Some NNs are models of biological neural networks
and some are not
Overview (Contd.)
• Historically, much of the inspiration for the field of NNs
came from the desire to produce artificial systems
capable of
• sophisticated, perhaps intelligent, computations similar to
those that the human brain routinely performs, and thereby
possibly to enhance our understanding of the human brain.
Overview (Contd.)
• Most NNs have some sort of training rule. In other words, NNs learn
from examples
• as children learn to recognize dogs from examples of dogs) and exhibit some
capability for generalization beyond the training data
• Neural computing must not be considered as a competitor to
conventional computing.
• Rather should be seen as complementary
• Most successful neural solutions have been those which operate in conjunction with
existing, traditional techniques.
Overview (Contd.)
Digital Computers
• Deductive Reasoning. We apply
known rules to input data to
produce output
• Computation is centralized,
synchronous, and serial.
• Memory is packetted, literally
stored, location addressable
• Not fault tolerant. One transis-
tor goes and it no longer works.
• Exact.
• Static connectivity.
• Applicable if well defined rules
with precise input data.
Neural Networks
• Inductive Reasoning. Given input
and output data (training
examples), we construct rules
• Computation is collective,
asynchronous, and parallel.
• Memory is distributed,
internalized, short term and
content addressable.
• Fault tolerant, redundancy, and
sharing of responsibilities.
• Inexact.
• Dynamic connectivity.
• Applicable if rules are unknown
or complicated, or if data are
noisy or partial.
Why Neural Networks
• Adaptive learning
• An ability to learn how to do tasks based on the data given for
training or initial experience.
• Self-Organization
• An ANN can create its own organization or representation of the
information it receives during learning time.
• Real Time Operation
• An ANN computations may be carried out in parallel, and special
hardware devices are being designed and manufactured which
take advantage of this capability.
• Fault Tolerance via Redundant Information Coding:
• Partial destruction of a network leads to the corresponding
degradation of performance. However, some network capabilities
may be retained even with major network damage.
What can you do with an NN
and what not?
• In principle, NNs can compute any computable function,
i.e., they can do everything a normal digital computer can
do.
• In practice, NNs are especially useful for classification and
function approximation problems.
• NNs are, at least today, difficult to apply successfully to
problems that concern manipulation of symbols and
memory.
• There are no methods for training NNs that can magically
create information that is not contained in the training
data.
Who is concerned with NNs?

• Computer scientists want to find out about the properties of

non-symbolic information processing with neural nets and
about learning systems in general.
• Statisticians use neural nets as flexible, nonlinear regression
and classification models.
• Engineers of many kinds exploit the capabilities of neural
networks in many areas, such as signal processing and
automatic control.
• Cognitive scientists view neural networks as a possible
apparatus to describe models of thinking and consciousness
(High-level brain function).
• Neuro-physiologists use neural networks to describe and
explore medium-level brain function (e.g. memory, sensory
system, motorics).
Who is concerned with NNs?

• Physicists use neural networks to model phenomena in

statistical mechanics and for a lot of other tasks.
• Biologists use Neural Networks to interpret nucleotide
sequences.
• Philosophers and some other people may also be
interested in Neural Networks for various reasons
Biological inspiration
• Animals are able to react adaptively to changes in their
external and internal environment, and they use their
nervous system to perform these behaviours.
• An appropriate model/simulation of the nervous system
should be able to produce similar responses and
behaviours in artificial systems.
• The nervous system is build by relatively simple units, the
neurons, so copying their behavior and functionality should
be the solution.
Biological inspiration (Contd.)
Biological inspiration (Contd.)
• The brain is a collection of about 10 billion interconnected
neurons
• Each neuron is a cell that uses biochemical reactions to receive,
process and transmit information.
• Each terminal button is connected to other neurons across
a small gap called a synapse
• A neuron's dendritic tree is connected to a thousand
neighbouring neurons
• When one of those neurons fire
• a positive or negative charge is received by one of the dendrites.
The strengths of all the received charges are added together
through the processes of spatial and temporal summation.
Artificial neurons
Neurons work by processing information. They receive and
provide information in form of spikes.
x1
x2 w1
n Output
x3 w2 z   wi xi ; y  H ( z )
Inputs

i 1 y
w3
… ..
. w
xn-1 n-1

xn wn
The McCullogh-Pitts model
Artificial neurons
Nonlinear generalization of the McCullogh-Pitts
neuron:
y  f ( x, w)
y is the neuron’s output, x is the vector of inputs, and w
is the vector of synaptic weights.
Examples: 1
y T
w xa
sigmoidal neuron
1 e
|| x  w||2

2a 2 Gaussian neuron
ye
 Activation Functions

• The activation function is generally non-linear

• Linear functions are limited because the output is simply proportional to the input
Activation Functions (Contd.)
Artificial neurons
Nonlinear generalization of the McCullogh-Pitts
neuron:
y  f ( x, w)
y is the neuron’s output, x is the vector of inputs, and w
is the vector of synaptic weights.
Examples: 1
y T
w xa
sigmoidal neuron
1 e
|| x  w||2

2a 2 Gaussian neuron
ye
Neural networks offer the following useful
properties and capabilities:
1. Nonlinearity. An artificial neuron can be linear or nonlinear. A neural
network, made up of an interconnection of nonlinear neurons, is itself
nonlinear
2. Input–Output Mapping. A popular paradigm of learning, called
learning with a teacher, or supervised learning, involves modification of
the synaptic weights of a neural network by applying a set of labeled
training examples, or task examples
3. Adaptivity. Neural networks have a built-in capability to adapt their
synaptic weights to changes in the surrounding environment
4 .Evidential Response. In the context of pattern classification, a neural
network can be designed to provide information not only about which
particular pattern to select, but also about the confidence in the
decision made
5. Contextual Information. Knowledge is represented by the very
structure and activation state of a neural network. Every neuron in the
network is potentially affected by the global activity of all other
neurons in the network. Consequently, contextual information is dealt
with naturally by a neural network.
6. Fault Tolerance. A neural network, implemented in hardware form,
has the potential to be inherently fault tolerant, or capable of robust
computation, in the sense that its performance degrades gracefully
under adverse operating conditions.
For example, if a neuron or its connecting links are damaged, recall of a
stored pattern is impaired in quality
7. VLSI Implementability. The massively parallel nature of a neural
network makes it potentially fast for the computation of certain tasks.
This same feature makes a neural network well suited for
implementation using very-large-scale-integrated (VLSI) technology.
8. Uniformity of Analysis and Design. Basically, neural networks enjoy
universality as information processors. We say this in the sense that
the same notation is used in all domains involving the application of
neural networks. This feature manifests itself in different ways:
• Neurons, in one form or another, represent an ingredient common to
all neural networks.
• This commonality makes it possible to share theories and learning
algorithms in different applications of neural networks.
• Modular networks can be built through a seamless integration of
modules.
9. Neurobiological Analogy. The design of a neural network is
motivated by analogy with the brain, which is living proof that fault-
tolerant parallel processing is not only physically possible, but also fast
and powerful.

• Supervised Learning: Modifying the synaptic weights by applying a set

of training samples, which constitute of input signals and
corresponding desired responses.
Neural Networks
Neuron
Neuron Diagram
Bias of Neuron
Neuron Models
Step Function
Ramp Function
Sigmoid Function
Gaussian Function is the Probability Function of Normal
Distribution.
Sometimes also called frequency curve
Network Architectures
Single Layer Feed-Forward
 Perceptron: Neural Model
(Special form of Single Layer feed forward)
Perceptron of Classification
 ‫الشبكات العصبية االصطناعية‬
‫الجهاز البشرى لل‪B‬معالجة يقوم أساسا على الخاليا العصبية في المخ و التي تعتمد على ا‪B‬لخلية العصبية ‪ Neuron‬كوحدة بنائية لها و‬
‫يبين الشكل تمثيل مبسط للخلية العصبية حيث يتكون التركيب من جسم الخلية ‪ Soma‬و الل‪B‬يف العصبى لمدخل الخلية ‪Dendrites‬‬
‫و الذى يحمل لالشارات من المخارج لخاليا عصبية أخرى الى مداخل هذه الخلية عن طريق مشابك‪ Synapes 4‬للتوصيل ثم الليفة‬
‫العصبية الوحيدة للمخرج ‪ Axon‬و التي تحمل ا‪B‬لنبضة الخارجة من الخلية الى خاليا أخرى متعددة‪.‬‬
 ‫• لقد تم تطوير نموذج عنصر حسابى ‪ Processing Element‬مكافئ لهذا النيرون و‬
‫الموضح بالشكل السابق و الذى يمثل عنصر المعالجة الذى يقوم بعمل محاكاة للنيرون‬
‫الطبيعى من تجميع لالشارات الموزنة عند المدخل ثم مقارنة المجموع بقيمة حدية داخلية‬
‫‪ Threshold‬حيث يعطى عنصر المعالجة نبضة في المخرج اذا زاد المجموع عن القيمة‬
‫الحدية و ال يعطى أية نبضات اذا كانت اقل منها و لقد تم استخدام عناصر المعالجة في بناء‬
‫الشبكات االصطناعية‪.‬‬
‫• ان الشبكات العصبية االصطناعية ‪ Artificial Neural Networks‬هي تركيبات للمعالجة‬
‫المتوازية الموزعة ‪ Parallel Distributed Processing Structure‬تعتمد اساسا على‬
‫عنصر المعالجة ‪ Processing Element‬القادر على العمل كذاكرة محلية ‪Local‬‬
‫‪ Memory‬مع اجراء العمليات المختلفة و الذى له مخرج واحد يتفرع الى كثير من التفرعات‬
‫التى تحمل نفس االشارة الخارجة منه مع بقاء المعالجة محلية أى انها تعتمد على القيم المدخلة‬
‫و كذلك القيم المخزونة بالذاكرة المحلية لهذه العناصر الحسابية‪.‬‬
 ‫• يمكن وضع تصور فلسفى آخر لتعريف الشبكات العصبيةاالصطناعية و ذلك بوصفها على‬
‫انها رسوم بيانية موجهه ‪ Directive Graph‬ذات حواف موزنة ‪ Weighted Edges‬قادرة‬
‫على تخزين األشكال و البصمات ‪ Patterns‬و ذلك بتعديل قيم األوزان للحواف‪.‬‬
‫• و بذلك يمكن لهذه الشبكات ان تتعرف على هذه البصمات أو األشكال مرة ثانية اذا كانت القيم‬
‫المدخلة غير كاملة ‪ Incomplete‬أو غير معرفة ‪Unknown‬‬
‫• تشترك معظم الشبكات العصبية فيما يلى‪:‬‬
‫‪ -1‬التمثيل الموزع ‪Distribution Representation‬‬
‫‪ -2‬المعالجة المحلية ‪Local Processing‬‬
‫‪ -3‬المعالجة الالخطية ‪Nonlinear Processing‬‬
‫يبين الشكل التالى امثلة من المعماريات المختلفة من الشبكات‬
‫المعماريات المختلفة للشبكات‬