This action might not be possible to undo. Are you sure you want to continue?

BooksAudiobooksComicsSheet Music### Categories

### Categories

### Categories

Editors' Picks Books

Hand-picked favorites from

our editors

our editors

Editors' Picks Audiobooks

Hand-picked favorites from

our editors

our editors

Editors' Picks Comics

Hand-picked favorites from

our editors

our editors

Editors' Picks Sheet Music

Hand-picked favorites from

our editors

our editors

Top Books

What's trending, bestsellers,

award-winners & more

award-winners & more

Top Audiobooks

What's trending, bestsellers,

award-winners & more

award-winners & more

Top Comics

What's trending, bestsellers,

award-winners & more

award-winners & more

Top Sheet Music

What's trending, bestsellers,

award-winners & more

award-winners & more

Welcome to Scribd! Start your free trial and access books, documents and more.Find out more

Chapter 1 Dr. Adnan Shaout

Introduction

networks were started about 50 years ago. Their early abilities were exaggerated, casting doubts on the field as a whole. There is a recent renewed interest in the field, however, because of new techniques and a better theoretical understanding of their capabilities.

Neural

**Motivation for neural networks
**

Scientists

are challenged to use machines more effectively for tasks currently solved by humans. Symbolic Rules don't reflect processes actually used by humans Traditional computing excels in many areas, but not in others.

Types of Applications

Machine learning: Having a computer program itself from a set of examples so you don't have to program it yourself. Optimization: given a set of constraints and a cost function, how do you find an optimal solution? E.g. traveling salesman problem. Classification: grouping patterns into classes: i.e. handwritten characters into letters. Associative memory: recalling a memory based on a partial match. Regression: function mapping

Application ± Cont. Cognitive science: Modelling higher level reasoning: ± language ± problem solving Modelling lower level reasoning: ± vision ± audition speech recognition ± speech generation .

Cont. Philosophy: Can human souls/behavior be explained in terms of symbols. etc. or does it require something lower level. like a neurally based model? . Overlaps with cognitive folks. Neurobiology: Modelling models of how the brain works: neuron-level neuron higher levels: vision. Mathematics: Nonparametric statistical analysis and regression. hearing.

navigation. e. Robotics .g. and match with the dock get converted to steering instructions. Siemens successfully uses neural networks for process automation in basic industries.g. rear position. i. recognizing handwritten characters.e. e. with adaptive echo canceling. . the current version of Apple's Newton uses a neural net.g..Where are neural networks being used? Signal processing: suppress line noise. vision recognition Pattern recognition. Manufacturing plants for controlling automated machines. blind source separation Control: e. in rolling mill control more than 100 neural networks to do their job. 24 hours a day. backing up a truck: cab position.

. Financial Applications: time series analysis. Medicine. image. . chess. go. visual search engines Business. e.g.g.e. e. stock market prediction Data Compression: speech signal. resulting in a network that has a high level of agreement with human experts. faces Game Playing: backgammon.. . i. rules for mortgage decisions are extracted from past decisions made by experienced evaluators. storing medical records based on case information Speech production: reading text aloud (NETtalk) Speech recognition Vision: face recognition . edge detection..

Hence pattern-directed retrieval and matching patternoperations are promoted .Why Neural Nets? Nets? The autonomous local processing of each individual unit combines with similar simple behavior of many other units to produce "interesting. Robust computation because knowledge is distributed and continuous rather than discrete or digital. Knowledge captured in a large number of fine-grained fineunits and can match noisy and incomplete data Fault tolerant architecture because computations can be organized so as not to depend on a fixed set of units and connections . global behavior Intelligent behavior is an "emergent" phenomenon Solving problems using a processing model that is similar to the brain may lead to solutions to complex information processing problems that would be difficult to achieve using traditional symbolic approaches in AI Associative memory access is directly represented." complex.

Neurobiology Constraints on Human Information Processing of neurons: 1011 Number of connections: 104 per neuron Neuron death rate: 105 per day Neuron birth rate: 0 Connection birth rate: very slow Performance: about 102 msec. or about 100 sequential neuron firings for "many" tasks Number .

....... distributed information processing High degree of connectivity among basic units Connections are modifiable based on experience Learning is a constant process...Properties of Nervous Systems Parallel.. and usually unsupervised Learning is based only on local information Performance degrades gracefully if some units are removed etc.

during which the neuron period. the neuron discharges a spike . ± Dendrites (inputs) ± Cell body ± Axon (output) A neuron receives input from other neurons (typically many thousands). down the axon.Neurons and Synapses The basic computational unit in the nervous system is the nerve cell. is unable to fire. to the next neuron(s) (or other receptors) . . This spiking event is also called depolarization. or neuron. Inputs sum (approximately). Once input exceeds a critical level. and is depolarization. followed by a refractory period.an electrical pulse that travels from the body. A neuron has: neuron.

amount of neurotransmitter reabsorbed. This link is called a synapse. the number and arrangement of receptors. Transmission of an electrical signal from one neuron to the next is effected by neurotransmitters. synapse. released from the first neuron and which bind to receptors in the second. chemicals which are neurotransmitters.g. The extent to which the signal from one neuron is passed on to the next depends on many factors. . e.The axon endings (Output Zone) almost touch the dendrites or cell body of the next neuron. etc. the amount of neurotransmitter available.

they learn "on-line". based "onon experience. one way brains learn is by altering the strengths of connections between neurons. From what we know of neuronal structures.Synaptic Learning Brains learn. and by adding or deleting connections between neurons. Of course. . and typically without the benefit of a benevolent teacher (But of cores by the well of Alaah). Furthermore.

Cont. One way this happens is through release of more neurotransmitter. Many other changes may also be involved . providing both memory and learning through long-term Potentiation. The efficiency of a synapse can change as a result of experience. longPotentiation.

LongLong-term Potentiation (LTP): An enduring (>1 hour) increase in (>1 synaptic efficacy that results from highhighfrequency stimulation of an afferent (input) pathway Hebbs Postulate: "When an axon of cell A excites cell B and repeatedly or persistently takes part in firing it." . some growth process or metabolic change takes place in one or both cells so that A's efficiency as one of the cells firing B is increased.

Bliss and Lomo discovered LTP in the hippocampus in 1973 Points to note about LTP: ± Synapses become more or less important over time (plasticity) ± LTP is based on experience ± LTP is based only on local information (Hebb's postulate) .Cont.

.

.

. distributed information processing ± high degree of connectivity among basic units ± connections are modifiable based on experience ± learning is a constant process...Summary The following properties of nervous systems: ± parallel... and usually unsupervised ± Learning is based only on local information ± performance degrades gracefully if some units are removed ± etc. .....

Each neuron also has a threshold value. For example. The inputs each have a weight that they contribute to the neuron. The sum of the input's weights is 0. the neuron is off. The only condition which would activate this neuron is if the top input were active and bottom one were inactive. neurons in the brain can have as many as a thousand inputs. If the sum of all the weights of all active inputs is greater than the threshold. Since 0 is smaller than 0. then the neuron is active. consider the case where both inputs are active. it may be simulated by a very simple model. . if the input is active.5. The neuron can have any number of inputs.A Model of a Neuron As complicated as the biological neuron is. This single neuron and its input weighting performs the logical expression A and not B .

Model of an artificial neuron .

wn are real-valued realparameters called weights net = w1 x1 + w2 x2 +«+ wn xn is called the weighted sum f: R R is called the activation function y = f(net) is the output of the neuron 5.. 4. 2. x2. .Model of an artificial neuron Terminology 1.. .. . 3.... xn are the inputs to the neuron w1.. w2. x1..

Model of an artificial neuron .

Examples of activation functions .

Identity Function The identity function is given by .

Step Function f(x) = -1 if x 0 = 1 if x > 0 .

If g(x) is the standard sigmoid then the symmetric sigmoid is .Symmetric Sigmoid The symmetric sigmoid is simply the sigmoid that is stretched so that the y range is 2 and then shifted down by 1 so that it ranges between -1 and 1.

Gaussian.Radial Basis Functions A radial basis function is simply a . .

± Linear threshold unit (LTU) used at output layer node(s) .e. one input layer and one output layer. output layer consists of just one unit . In most basic form.i.Perceptrons Simplest "interesting" class of neural networks ± ³1 layer" network -..

.. and t is an extra weight called the bias . + wn*xn > t where t is a *x1 w2*x2 threshold value. in an implementation. *x2 t*(So. .. by definition of an LTU. output of a unit is defined by w1*x1 + w2*x2 + .. Now each unit has a fixed threshold value of 0. Threshold associated with LTUs can be considered as another weight. + wn*xn + t*(-1) > 0 . But this is algebraically equivalent to w1*x1 + *x1 w2*x2 + . consider each LTU as having an extra input which has a constant input value of -1 and the arc's weight is t . That is.

Threshold Logic Gate Definition .A threshold logic gate (LTG) is an artificial neuron where the activation function is taken to be a threshold function .

LTG Question What can we do with these artificial neurons? Answer We can compute stuff! Claim LTG¶ LTG¶s can operate as logic gates .

Linear Threshold Gates Exercise Choose values of w1.For (a) Draw truth table (b) K-Map K(c) digital circuit . and w3 (for the bias w1 w2 w3 input) to realize the AND function. w2. Conclusion We can build a computer with LTGs! Example .

5. 0...5 with t=0.5] [0 75] [.25 t=0 .5.5. choose W = [0.5 with t=0. To realize the AND function.75] or W = [.5.5. 0.5.5] [0 25] [.5.. 0.25] or W = [..75 t=0 To realize the OR function.5. choose W = [0. 0.

Switching Functions Claim This switching function can be realized with a single LTG Proof Consider .

Hint: construct a truth table of the form: .Exercise ±Cont. Determine the Boolean function realized.

an example is a list of values defining the values of the input units in a given network. I.e. O. e) neural-nete) T = desired (i.e. T ) Note: Each pass through all of the training examples is called one epoch .Learning in Neural Nets Programmer specifies numbers of units in each layer and connectivity between units. so the only unknown is the set of weights associated with the connections Supervised learning of weights from a set of training examples. teacher) output updateupdate-weights(e.. The output of the network is a list of values output by the output units Algorithm : ± Initialize the weights in the network (usually with random values ( ± repeat until stopping criterion is met ± for each example e in training set do O = neural-net-output(network. given at I/O pairs.

.O) xi ± xi is the input associated with the ith input unit . ± alpha is a constant between 0 and 1 called the learning rate .Perceptron Learning Rule In a Perceptron. we define the updateupdate-weights function in the learning algorithm above by the formula : ± wi = wi + delta_wi ± where delta_wi = alpha * (T .

. the error is monotonically non-increasing ( non Correct output (T = O) causes no change in a weight xi 0 = causes no change in weight Does not depend on wi If T=1 and O=0. then decrease the weight so that hopefully T=0 O=1 next time the result will be below the threshold and cause the output to be 0 . A product is used as a means of measuring the correlation between the values output by the two units . this rule will be used to iteratively adjust all of the weights so that at each iteration (training example) the error is decreasing (more correctly. Also called the Delta Rule or the Widrow-Hoff Rule Widrow "Local" learning rule in that only local information in the network is needed to update a weight Performs gradient descent in "weight space" in that if there are n weights in the network. then increase the weight so that hopefully T=1 O=0 next time the result will exceed the threshold at the output unit and cause the output O to be 1 If T=0 and O=1.Notes about this update formula : Based on a basic idea due to Hebb that the strength of a connection between two units should be adjusted in proportion to the product of their simultaneous activations.

Example: Learning OR in a Perceptron Given initial network defined as: Let the learning rate parameter be alpha = 0. w3 with constant input w3 value x3 = -1 x3 .2 Let the threshold be specified as a third weight.

7 .3 .7 .4 .2 0 0 0 0 0 .2 .1 .2 .5 .5 .7 .5 .5 0 .5 .8 .2 .4 .2 .2 .5 .7 .5 .7 .2 0 0 0 -.The result of executing the learning algorithm for 3 epochs: x1 x2 T O delta_w1 w1 delta_w2 w2 delta_w3 w3 (=t) 0 0 1 1 0 0 1 1 0 0 1 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 1 1 0 1 1 1 0 1 1 1 0 0 0 1 0 1 0 1 0 1 1 1 0 0 .7 .7 .2 .2 -.1 .4 .8 .2 0 0 0 0 0 0 0 0 0 0 .7 .3 .4 .2 0 0 0 .3 .7 0 -.2 0 0 0 0 0 .6 .3 .7 .7 .5 .1 .

So. the final learned network is : .

Are you sure?

This action might not be possible to undo. Are you sure you want to continue?

We've moved you to where you read on your other device.

Get the full title to continue

Get the full title to continue listening from where you left off, or restart the preview.

scribd