Contents

Abbreviations and Symbols
1 Introduction
1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 1.10
What Is a Neural Network? Structural Levels of Organization in the Brain Models of a Neuron Neural Networks Viewed as Directed Graphs Feedback Network Architectures Knowledge Representation Visualizing Processes in Neural Networks Artificial Intelligence and Neural Networks Historical Notes Problems

xvi i
1
1 6 8 13 15 18 22
31

32
36 41

2 Learning Process
2.1 2.2 2.3 2.4 2.5 2.6 2.7
Introduction Error-Correction Learning Hebbian Learning Competitive Learning Boltzmann Learning The Credit-Assignment Problem Supervised Learning

45
45 47 49 53 55 56 57
ix

x Contents

28 . 29 . 21 .0 2.11 21 .2 21 .3 21 .4

Reinforcement Learning Unsupervised Learning Learning Tasks Adaptation and Learning Statistical Nature of the Learning Process Learning Theory Discussion Problems

59 65 66 69 71 75 85 87

3 Correlation Matrix Memory
31 . 32 . 33 . 34 . 35 .
Introduction Distributed Memory Mapping Correlation Matrix Memory Error Correction Applied to a Correlation Matrix Memory Discussion Problems

90
90 92 95 100 102 104

4 The Perceptron
41 . 42 . 43 . 44 . 45 . 46 .
Introduction Basic Considerations The Perceptron Convergence Theorem Performance Measure Maximum-Likelihood Gaussian Classifier Discussion Problems

106
106 107 108 113 115 118 120

5 Least-Mean-Square Algorithm
51 Introduction . 5 2 Wiener-Hopf Equations . 53 Method of Steepest Descent .

121
121 122 124

Contents xi

5.4 Least-Mean-Square Algorithm 5.5 Convergence Considerations of the LMS Algorithm 5.6 Learning Curve 5.7 Learning-Rate Annealing Schedules 5.8 Adaline 5.9 Discussion Problems

126 129 131 132 134 135 136

6 Multilayer Perceptrons
6.1 6.2 6.3 6.4 6.5 6.6 6.7
Introduction Some Preliminaries Derivation of the Back-Propagation Algorithm Summary of the Back-Propagation Algorithm Initialization The XOR Problem Some Hints for Making the Back-Propagation Algorithm Perform Better 6.8 Output Representation and Decision Rule 6.9 Computer Experiment 6.10 Generalization 6.11 Cross-Validation 6.12 Approximations of Functions 6.13 Back-Propagation and Differentiation 6.14 Virtues and Limitations of Back-Propagation Learning 6.15 Accelerated Convergence of Back-Propagation Through Learning-Rate Adaptation 6.16 Fuzzy Control of Back-Propagation Learning 6.17 Network-Pruning Techniques 6.18 Supervised Learning Viewed as a Nonlinear Identification Problem 6.19 Supervised Learning as a Function Optimization Problem 6.20 Supervised Learning of Probability Distributions by Multilayer Perceptrons 6.21 Discussion

138
138 140 142 153 156 157 160 162 165 176 179 181 185 186 192 201 205 212 215 217 220

xii Contents

6.22 Applications Problems

22 1 229

7 Radial-Basis Function Networks
7.1 7.2 7.3 7.4 7.5 7.6 7.7 7.8 7.9 7.10 7.11 7.12 7.13 7.14 7.15
Introduction Cover’s Theorem on the Separability of Patterns Interpolation Problem Supervised Learning as an Ill-Posed Hypersurface Reconstruction Problem Regularization Theory Regularization Networks Generalized Radial-Basis Function Networks The XOR Problem (Revisited) Comparison of RBF Networks and Multilayer Perceptrons Mixture Models Learning Strategies Computer Experiment Factorizable Radial-Basis Functions Discussion Applications Problems

236
236 237 242 244 245 255 256 260 262 263 264 268 269 272 274 28 1

8 Recurrent Networks Rooted in Statistical Physics 285
8.1 8.2 8.3 8.4 8.5 8.6
Introduction Dynamical Considerations The Hopfield Network Computer Experiment I Energy Function Error Performance of the Hopfield Network

285 287 289 296 300 303 308

8.7 Isomorphism Between a Hopfield Network and a SpinGlass Model

Contents xiii

8.8 Stochastic Neurons 8.9 Phase Diagram of the Hopfield Network, and Related Properties 8.10 Simulated Annealing 8.11 The Boltzmann Machine 8.12 A Markov Chain Model of the Boltzmann Machine 8.13 The Mean-Field-Theory Machine 8.14 Computer Experiments I1 8.15 Discussion Problems

309 311 3 14 318 330 335 341 344 346

9 Self-organizing Systems I: Hebbian Learning
9.1 9.2 9.3 9.4 9.5 9.6 9.7 9.8
Introduction Some Intuitive Principles of Self-organization Self-organized Feature Analysis Discussion Principal Components Analysis A Linear Neuron Model as a Maximum Eigenfilter Self-organized Principal Components Analysis Adaptive Principal Components Analysis Using Lateral Inhibition 9.9 Two Classes of PCA Algorithms 9.10 How Useful Is Principal Components Analysis? Problems

352
352 353 354 362 363 370 374 382 39 1 392 394

10 Self-organizing Systems 11: Competitive Learning
10.1 Introduction 10.2 Computational Maps in the Cerebral Cortex 10.3 Two Basic Feature-Mapping Models 10.4 Modification of Stimulus by Lateral Feedback

397
397 398 400 402
,

xiv Contents

10.5 Self-organizing Feature-Mapping Algorithm 10.6 Properties of the SOFM Algorithm 10.7 Reformulation of the Topological Neighborhood 10.8 Adaptive Pattern Classification 10.9 Learning Vector Quantization 10.10 Applications 10.11 Discussion
Problems

408 414 422 424 427 430 434 434

11 Self-organizing Systems II I: Information444 Theoretic Models
11.1 Introduction 11.2 Shannon’s Information Theory 11.3 The Principle of Maximum Information Preservation 11.4 Generation of Topologically Ordered Maps 11.5 Discussion 11.6 Spatially Coherent Features 11.7 Another Information-Theoretic Model of the Perceptual System 11.8 Concluding Remarks
Problems

444 445 452 458 46 1 464 469 47 1 47 1

12 Modular Networks
12.1 Introduction 12.2 Basic Notions of Modularity 12.3 Associative Gaussian Mixture Model 12.4 Stochastic-Gradient Learning Algorithm 12.5 Hierarchical Structure of Adaptive Expert Networks 12.6 Piecewise Control Using Modular Networks

473
473 475 478 482 487 49 1

Contents xv

12.7 Summary and Discussion Problems

493 495

13 Temporal Processing
13.1 13.2 13.3 13.4 13.5 13.6 13.7 13.8
Introduction Spatio-Temporal Models of a Neuron FIR Multilayer Perceptron Temporal Back-Propagation Learning Temporal Back-Propagation with Adaptive Time Delays Back-Propagation Through Time Real-Time Recurrent Networks Real-Time Nonlinear Adaptive Prediction of Nonstationary Signals 13.9 Partially Recurrent Network 13.10 Discussion Problems

498
498 501 506 5 10 518 520 521 526 531 532 533

14 Neurodynamics
14.1 14.2 14.3 14.4 14.5 14.6 14.7 14.8 14.9 14.10 14.11 14.12 14.13
Introduction Dynamical Systems Stability of Equilibrium States Attractors Strange Attractors and Chaos Neurodynamical Models Manipulation of Attractors as a Recurrent Network Paradigm Dynamics of Hopfield Models The Cohen-Grossberg Theorem The Hopfield Model as a Content-Addressable Memory Brain-State-in-a-Box Model Recurrent Back-Propagation Discussion Problems

537
537 539 542 548 550 551 555 556 56 1 563 57 1 577 585 589

xvi Contents

15 VLSl Implementations of Neural Networks
15.1 Introduction 15.2 Major Design Considerations 15.3 Categories of VLSI Implementations 15.4 Neurocomputing Hardware 15.5 Concluding Remarks Problems

593
593 594 596 602 612 615

Appendix A

Pseudoinverse Matrix Memory

617

Appendix B A General Tool for Convergence Analysis of Stochastic Approximation Algor it hms 619 Appendix C Statistical Thermodynamics Appendix D Fokker-Planck Equation Bibliography Index
625 628 635 691

Sign up to vote on this title
UsefulNot useful