This action might not be possible to undo. Are you sure you want to continue?

Contents

PREFACE ACKNOWLEDGMENTS CHAPTER 1 INTRODUCTION

xiii xv,

1

1.1 1.2

**Why Neural Networks, and Why Now? What Is a Neural Net? 3
**

1.2.1 1.2.2 Artificial Neural Networks, 3 Biological Neural Networks, 5 Signal Processing, 7 Control, 8 Pattern Recognition, 8 Medicine, 9 Speech Production, 9 Speech Recognition, 10 Business, 11

1

1.3

**Where Are Neural Nets Being Used? 7
**

1.3.1 1.3.2 1.3.3 1.3.4 1.3.5 1.3.6 1.3.7

1.4

**How Are Neural Networks Used?
**

1.4.1 1.4.2 1.4.3 1.4.4

11

Typical Architectures, 12 Setting the Weights, 15 Common Activation Functions, 17 Summary of Notation, 20

vii

Preface

There has been a resurgence of interest in artificial neural networks over the last few years, as researchers from diverse backgrounds have produced a firm theoretical foundation and demonstrated numerous applications of this rich field of study. However, the interdisciplinary nature of neural networks complicates the development of a comprehensive, but introductory, treatise on the subject. Neural networks are useful tools for solving many types of problems. These problems may be characterized as mapping (including pattern association and pattern classification), clustering, and constrained optimization. There are several neural networks available for each type of problem. In order to use these tools effectively it is important to understand the characteristics (strengths and limitations) of each. This book presents a wide variety of standard neural networks, with diagrams of the architecture, detailed statements of the training algorithm, and several examples of the application for each net. In keeping with our intent to show neural networks in a fair but objective light, typical results of simple experiments are included (rather than the best possible). The emphasis is on computational characteristics, rather than psychological interpretations. TO illustrate the similarities and differences among the neural networks discussed, similar examples are used wherever it is appropriate. Fundamentals of Neural Networks has been written for students and for researchers in academia, industry, and govemment who are interested in using neural networks. It has been developed both as a textbook for a one semester, or two quarter, Introduction to Neural Networks course at Florida Institute of Technology, and as a resource book for researchers. Our course has been developed jointly by neural networks researchers from applied mathematics, comxiii

the mathematical derivation of the backpropagation training algorithm makes clear the correct order of the operations. are also used for these types of problems. Our students are seniors. can logically follow chapter 2. The order of presentation of the topics was chosen to reflect increasing complexity of the networks. are two of the earliest applications of neural networks with adaptive weights. and 7. including two for constrained optimization problems. Chapters 4 and 5 treat networks for clustering problems (and mapping networks that are based on these clustering networks). SO that the chapters (after the first chapter) may be used in almost any order desired. discussed in later chapters. Chapter 6. More complex networks. Algorithms. 6. rather than computer codes. covered in chapters 2 and 3. It is assumed that the reader is familiar with calculus and some vector-matrix notation and operations. consistent with the primary aims of clarity and correctness. rather than fostering the more superficial familiarity that sometimes results from using completely developed software packages. and computer and electrical engineering. The material in each chapter is largely independent. Derivations. are provided to encourage the reader to develop a thorough understanding of the mechanisms of training and applying the neural network. The level of mathematical sophistication increases somewhat in the later chapters. theorems and proofs are included when they serve to illustrate the important features of a particular neural network. For many applications. in science and engineering. For example. SO that they can be skipped without loss of continuity. the formulation of the problem for solution by a neural network (and choice of an appropriate network) requires the detailed understanding of the networks that cornes from performing both hand calculations and developing computer codes for extremely simple examples. . as is appropriate for the networks presented in chapters 5. Chapter 7 presents a few of the most widely used of the many other neural networks. derivations and proofs (when included) are presented at the end of a section or chapter. or graduate students. backpropagation.Xiv Preface puter science. Single layer nets for pattern classification and pattern association. The mathematical treatment has been kept at a minimal level. The McCulloch-Pitts neuron discussed at the end of Chapter 1 provides a simple example of an early neural net. However. although the networks in chapters 3-5 are somewhat simpler in structure. many work in local industry. as well as for more general mapping problems.

The illustrations of modified Hebbian learning have been adapted from the original pubxv . Laurie Walker assisted in the development of the backpropagation program for several of the examples in Chapter 6.12 and 4.11. Steven Rogers. My thanks go also to my colleagues for stimulating discussions and encouragement. Ti-Cheng Shih did the computations for Example 6. Alan Lindsay. and 5.8. John Karp provided the results for Example 4.Acknowledgments Many people have helped to make this book a reality. Ki-suck Yoo provided Example 7. 1 can only mention a few of them here. Fred Parker obtained the results shown in Examples 4. especially Harold K.5. and Tony Martinez. 4. Bernard Widrow. The spanning tree test data (Figures 4. Abdallah Said developed the logarithmic activation function used in Examples 6.9 and 4. 5.8.12) are used with permission from Springer-Verlag. Several of the network architecture diagrams are adapted from the original publications as referenced in the text. and Hong-wei Du assisted with the Boltzmann machine and Hopfield net examples in Chapter 7. Judith Lipofsky did Examples 4. and Mike Thursby. Joseph Oslakovic performed the computations for several of the examples in Chapter 5. Robin Schumann. Fred Ham.13. 1 have benefited either directly or indirectly from short courses on neural networks taught by Harold Szu.7 and 6.11. Todd Kovach. Robert Hecht-Nielsen. several of the examples are based on student work. Moti Schneider.10. Nazif Tepedelenlioglu. Barry Grossman. Joe Vandeville.12. Demetrios Lainiotis. My students have assisted in the development of this book in many ways.8. and Francisco Gomez performed the computations for many of the examples in Chapter 2. Brown.

Peter Anderson. and always. But first. Penn State University.xvi Acknowledgments lications: Figure 7. The diagrams of the ART2 architecture are used with permission of the Optical Society of America. encouragement. Palmer. The diagrams of the simple recurrent net for learning a context sensitive grammar (Servan-Schreiber. The Ohio State University.11 has been adapted and reprinted from Neural Networks. Don Fausett for introducing me to neural networks. et al. Vol. and Carpenter and Grossberg. Introduction to the Theory of Neural Computation. @ 1991 by Addison-Wesley Publishing Company. during the writing of this book (as well as other times). they are used with permission of IEEE. 441-457. UK. Modified Hebbian Leaming for Curve and Surface Fitting. 1989) are used with the permission of the authors. and Suen. . and especially Rick DeLorenzo. Inc. and advice when asked. Rochester Institute of Technology. and for his patience. Xu.. 1 thank Maurice Kurtz for making it available to me.. 1 appreciate the constructive and encouraging comments of the manuscript reviewers: Stanley Ahalt. and Nirmal Bose. et al. Several of the figures for the neocognitron are adapted from (Fukushima. pp.10 has been adapted from Hertz. 1 would like to thank the Prentice-Hall editorial staff. Oja. last. 5. Krogh. Oxford 0X3 OBW. Figure 7. for their diligent efforts to produce an accurate and attractive product within the inevitable time and budget constraints. The preparation of the manuscript and software for the examples has been greatly facilitated by the use of a Macintosh IIci furnished by Apple Computers under the AppleSeed project. Headington Hi11 Hall. 1 would like to thank my husband and colleague. 1983). 1992 with permission from Pergamon Press Ltd.

. OF FUNDAMENTALS NEURAL NETWORKS .

.

these do not necessarily reflect the actual process we use. together with some feedback from a “teacher. Even without a teacher. Based on examples. we can group similar patterns together. Early 1 .1. for example) while satisfying certain constraints (such as the need to earn a living). sequential. we may be able to describe rules by which we can make such decisions. More experience allows us to refine our responses and improve our performance. motivated by a desire to try both to understand the brain and to emulate some of its strengths. The development of artificial neural networks began approximately 50 years ago. but has been less successful for other types of problems.1 WHY NEURAL NETWORKS AND WHY NOW? As modern computers become ever more powerful. Traditional.” we learn easily to recognize the letter A or distinguish a cat from a bird. scientists continue to be challenged to use machines effectively for tasks that are relatively simple for humans. Each of these types of problems illustrates tasks for which computer solutions may be sought. Although eventually. logic-based digital computing excels in many areas. Yet another common human activity is trying to achieve a goal that involves maximizing a resource (time with one’s family.