1322
4, JULY 1996
Book Reviews
Fundamentals of Artificial Neural Networks-
M. H. Hassoun (Cambridge, MA: MIT Press, 1995)
Reviewer: Terrence L. Fine, Fellow, IEEE
Artificial neural networks are systems motivated by the distributed,
massively parallel computation in the brain that enables it to be so
successful at complex control and recognitiodclassification tasks. The
biological neural network that accomplishes this can be mathemat-
ically modeledcaricatured by a weighted, directed graph of highly
interconnected nodes (neurons). The artificial nodes are almost always
simple transcendental functions whose arguments are the weighted
summation of the inputs to the node; early work on neural networks
and some current work uses node functions taking on only binary
values. After a period of active development in the 1950’s and 1960’s,
that slowed in the face of the limitations of the networks then being
explored, neural networks experienced a renaissance in the 1980’s
with the work of Hopfield [7] on the use of networks with feedback
(graphs with cycles) as associative memories, and that of Rumelhart
et al. [13] on backpropagation training and feedforward (acyclic
graphs) networks that could “leam” from input-output examples
provided in a training set. Learning in this context is carried out
by a descent-based algorithm that adjusts the network weights so
that the network response closely approximates the desired responses
specified by the training set. This ability to learn from training data,
rather than needing to be explicitly (heuristically) programmed, was
important both for an understanding of the functioning of brains and
for progress in a great variety of applications in which practitioners
had been unable to embed their qualitative understanding in suc-
cessful programs. The capabilities of neural networks were quickly
exploited in a great number of applications to pattern classification,
control, and time-series forecasting. Hopfield’s work on associative
memories excited the interest of his fellow statistical physicists
who felt that their methods of analysis would be applicable and
productive in studying the asymptotic behavior of neural networks.
Unfortunately, many of the applications and studies were either trivial
or misguided and earned the field the sobriquet of “hype” for the
common appearance of exaggerated claims. Nonetheless, information
theory distinguished itself with such solid papers as that of McEliece
et al. [8] in providing mathematically sophisticated analyses of
network capabilities. The 1990’s saw a significant maturation both
in applications and in theoretical understanding of performance and
limitations. In particular, neural networks provided a wide spectrum
of applied statisticians with a new and powerful class of regression
and classification functions that, for the first time, allowed them
to make successful truly nonlinear models involving hundreds of
variables. The problem of “feature” or “regressor” selection becomes
less critical when you do not need to narrow your choices among
input variables. A new regime in statistics became accessible, and
applied statisticians were no longer restricted in practice either to
very simple nonlinear models in a few variables or to larger, but
linear, models based solely upon second-order properties.
Publisher Item Identifier S 0018-9448(96)05458-2
0018-9448/96$05.00 0 1996 IEEE
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 42, NO.
-
The initial generation of books on artificial neural networks that
appeared in the late 1980’s tended to be either highly simplified
overviews, with a significant emphasis on neurobiological issues, or
edited collections of papers, frequently with a physics orientation and
focus on Hopfield/recurrent/feedback networks. In the last couple
of years we have been fortunate to see the emergence of several,
engineering-oriented texts written by capable authors with systems
or statistics backgrounds. There continues to be a proliferation of
edited, and I use this term loosely, collections of papers serving
the ego-gratifying purpose of having conferences and workshops
appear in hard cover. Among the new generation of texts worth
study, the first was perhaps Hertz et al. [6] and those of Haykin
[5] and Zurada [16] are comparable to the current text by Hassoun
as being appropriate for electrical engineers. Haykin’s is the most
comprehensive of these books while Hassoun’s [4] is somewhat
more mathematical while also attempting to be comprehensive.
Unfortunately, a consequence of the attempts by all of these authors
to be comprehensive is that on many important topics their treatment
is too superficial for a good seniorhrst-year-graduate course on this
subject, and too superficial to please the readers of these Transactions.
Typically, mathematical results are quoted from other sources and
little or no supporting argument, let alone proofs, provided. An
attempt at wider communication that argues for a de-emphasis
on rigor becomes corrupted by a de-emphasis on precision and a
frequent absence of significant explanation and development. Those
seeking a deeper level of explanation will be interested in a newer
neural network literature represented by Ripley [ l l ] , Siu et al.
[14], and Vapnik [l5] that either make fewer compromises with
mathematical theory or explain mathematical issues more soundly.
Each of these monographs are more focussed in their treatment of
neural networks.
Hassoun’s goals are outlined in his Preface as, “emphasizes
fundamental theoretical aspects of the computational capabilities and
learning abilities of artificial neural networks . . . to present a unified
framework that makes the subject more accessible to students and
practitioners. . . . The main audience is first-year graduate students in
electrical engineering, computer engineering, and computer science.
. . . The theory and techniques . . . are fairly mathematical, although
the level of mathematical rigor is relatively low. . . . The operation
of artificial neural networks is viewed as that of nonlinear systems.”
The author, in a personal communication, has asserted that, “The
unifying framework is that one may treat all neural networks in terms
of an approximation model and an associated learning rule . . ..” This
claim of a unified framework is further amplified in the Preface by
the assertion, “a continuous-time learning rule is viewed as first-order
stochastic differential equatiorVdynamica1 system whereby the state
of the system evolves so as to minimize an associated instantaneous
criterion function.” While this viewpoint explains much of the order
of presentation in this text, it encompasses so much that it does not
define accurately the choice of topics. This is perhaps to be expected,
given the attempt to be comprehensive and the weight given to various
issues that are reflective of the author’s research interests. The low
level of mathematical rigor is less problematic than is the fact that
many mathematical results are simply quoted from the literature, a
failing common to this generation of neural network texts. Finally,
the target audience is an appropriate one for this text.
IEEE TRANSACTIONS ON INFORMATION THEORY. VOL. 42, NO. 4, JULY 1996
Chapter 1 and the first part of Chapter 2 contain about 45
pages devoted to neural networks with a binary-valued signum
function element called a linear threshold gate (LTG) and its easy
generalization to a polynomial threshold gate (PTG) in which the
inputs to the LTG are polynomial functions of the original inputs
rather than just the inputs themselves. These networks were the ones
originally envisioned by McCullough and Pitts in 1943 and studied
in some detail by Frank Rosenblatt [12]. Chapter 1 concentrates on
the properties of an individual linear threshold gate (step function)
and follows the seminal earlier work of Cover [3] and can be
compared to Nilsson [lo]. While Weierstrass’ Theorem on uniform
approximation to continuous functions by polynomials is cited in
Chapter 1, it is out of place in this discussion of approximation
by LTG’s and PTG’s; the synthesis of Boolean functions by PTG’s
requires far less than the Weierstrass Theorem. Results from older
studies on switching function realization are recalled, sometimes to
advantage and sometimes, as in the case of Karnaugh maps with
their applicability only to very small numbers of inputs, to little
advantage. However, it is of value to embed LTG-based networks
in their historical switching circuit predecessor. A good selection of
exercises rounds out Chapter 1, although several problems (e.g., ones
on polynomial approximation) have been given little foundation and
do not support the developments of the chapter.
Chapter 2 starts by analyzing the Boolean function representation
properties of LTG-based networks in terms of complexity of im-
plementations. More extensive discussion of such networks can be
found in the detailed treatment given by Siu et al. [I41 that is written
at the mathematical level of these Transactions. The approximation
properties of neural networks with real-valued nodes are noted, albeit
briefly, in Section 2.3. This is surprising given the detailed attention
given to LTG-based networks and the far greater importance in
practice of networks with real-valued nodes. Kolmogorov’s theorem
on representations of continuous functions of several variables is
presented in some detail even though it is largely irrelevant to
representations by neural networks. The essential results on universal
approximation properties of single hidden layer neural networks are
indicated in Section 2.3.2. Section 2.4.2 contains some speculations
on the energy required to perform computations that is of interest
but seems out of place. Chapter 2 closes with a small selection of
exercises that focus on the LTG case and not on the more important
real-valued case.
The major focus of this text, 220 pages of Chapters 3, 4, and 5,
is the issue of learning in neural networks. More than 20 supervised
(learning with classified data) and unsupervised learning algorithms
are introduced in Chapter 3. Chapter 3 opens with the perceptron
training algorithm of Rosenblatt, used to train a single binary-valued
node. Convergence is proven when the training set is learnable
(linearly separable) and some generalizations are given of the PTA.
The PTA is also derived as a gradient descent rule. There is a
brief discussion of the nonconvergent behavior of the PTA when
the training set is not linearly separable. More is known now about
this case and about the intrinsic difficulty of the nonseparable case
(e.g., Amaldi [l]). The widely discussed LMS rule is presented in
some detail. In all, there is about 30 pages of discussion of learn-
ingkraining for a single-node neural network, There is an extensive
presentation of unsupervised learning rules including Hebbian and
Linsker, principal components analysis, Kohonen vector quantization,
and Kohonen self-organizing feature maps. Hassoun notes that, “The
presentation of these rules is unified in the sense that they may all
be viewed as realizing incremental steepest-gradient-descent search
on a suitable criterion function.” Chapter 3 concludes with a nice
table summarizing characteristics of the wide variety of learning rules
discussed.
1323
Chapter 4 is an ambitious attempt to provide mathematical ana1,yses
of several of the preceding learning rules that address asymplotic
behavior as well as complexity and generalization performance. One
can question the impomrtance of asymptotic analyses that address
stability rather than analyses of statistical performance. Chapter 4
uses approximations of learning algorithms by ordinary differential
equations to study stability, and this is an area of research interest to
the author. I suspect that this approach has little practical importance
in terms of guidelines useful in deploying and controlling these
learning algorithms with finite training sets, limited computational
resources, and the prevalent use of training algorithms controlled to
limit the amount of training. Generalization is addressed on the bases
of both average and worst case performance analyses. The exposition
of average case generalization (e.g., eq. 4.8.4) does not meet the
standards of rigor expected by the statistical and information-theoretic
communities. The discwsion of worst case analysis is very brief and
amounts to a page of quotation primarily from Baum and Haussler [2].
The important issue of generalization ability of a trained network is
given short shrift compared to the less important issue of an idealized
asymptotic behavior of the training algorithm. The attached exercises
underscore this imbalance of treatment.
Chapter 5 treats the central topic of the backpropagation formu-
lation of steepest descent for training multilayer neural networks
in a somewhat thrown-together fashion. Curiously, Hassoun never
introduces a systematic notation for describing multilayer neural
networks. One is left with a diagram to indicate that w J Land ,wl3
denote weights connected to different layers. There is no thorough
discussion setting backpropagation in the context of steepest descent
algorithms. Indeed, Hassoun allows different learning rates for the
network output weights and for the hidden layer weights, and thereby
moves away from steepest descent without formally facing this
consequence. There is only a one-sided discussion of the relative
merits of on-line (called here “incremental”) versus batch processing,
with the recommendation solidly on the side of on-line processing
for its claimed advantage of being noisier and therefore possessing a
greater likelihood of not becoming trapped in a poor local minimum.
On-line processing can indeed be construed as more random in its
results than is batch processing. However, batch processing is the
only version that can implement true steepest descent. The choice
of fixed learning rate is addressed in a previous Section 4.2.2, where
bounds necessary for convergence are noted, and an “optimal learning
rate” is suggested in Seiztion 5.2.2.
Section 5.2 contains albout 25 pages on improvements to the basic
backpropagation algorithm presented in Section 5.1. Several impor-
tant training algorithms are touched on only briefly in Section 52.3,
entitled “Momentum.” Levenberg-Marquardt, the standard adopted
by the Matlab Neural Networks Toolbox, is inaccurately referred to in
one sentence on p.218. Industry standard quasi-Newton methods ,are
not noted, although some approximations to the Hessian appearing in
Newton methods are discussed briefly. The important idea of moving
from steepest descent search to conjugate gradient directions search is
not given the treatment it deserves. Conjugate gradient methods have
been quite successful in neural network applications (e.g., Moller
[9]). Weight decay and rtegularization methods are also deserving of
more discussion than that provided in Section 5.2.5. Cross-validation
is treated in only a qualitative fashion in Section 5.2.6. The chapter
concludes with a section containing a good variety of applications
culled from the literature: and a section that includes such topics as
time-delay neural networks (TDNN) and so-called backpropagation
through time used in controller design.
Hassoun reaches several general conclusions about the relative
values of networks with two hidden layers versus ones with a single
hidden layer, preferring networks with a single hidden layer to
1324
multiple-layer networks having the same number of weights. While
these conclusions are worth keeping in mind, too little is known
analytically about the relative merits of such networks to place much
confidence in them. Global optimization is an issue that Hassoun
examines in Chapter 8 as well as in Chapter 5. The suggestion made
in Chapter 5 is a revision of the loss or error function that is only
guaranteed to find a global minimum for networks with a single input
variable, an application far from typical for neural networks.
Chapter 6 treats a variety of other networks such as those based
on radial basis functions, the CMAC model, and so-called adaptive
resonance theory for clustering networks. Radial basis functions
will be familiar to some as the kernel methods used for estimating
probability density functions and for interpolation. The CMAC model,
while it has adherents, is peculiar in this setting for being a linear
mapping operating on features selected through Boolean operations
on the network inputs.
Chapter 7 treats associative memories in some detail. Models
include both linear systems and neural networks with feedback known
as Hopfield nets. While the analysis of the memory storage abilities
(capacity) of Hopfield nets had early contributions from information
theorists (e.g.. McEliece et al. [SI), the treatment provided herein will
disappoint readers of these Trunsactions for its brevity. Results in this
area are cited but neither derived nor discussed in detail. The chapter
concludes with a good variety of problems.
Chapter 8 devotes about 60 pages to such global optimization
methods for neural networks as simulated annealing and genetic
algorithms. While this i s a topic worth treating, one should not
make much of such approaches. Global optimization of functions of
hundreds or thousands of variables is an unrealistic demand that can
only be achieved if one devotes unlikely computational resources
to the optimization. Neural networks work in practice because a
judicious selection among several local minima turns out to be
sufficient.
There is an extensive bibliography of about 700 items covering the
period ending in 1993, and a good index. Each chapter closes with
a useful summary. In sum, Hassoun has written a worthy compre-
hensive text on a wide variety of neural networks and approaches to
their design that can, as he hoped, be the basis of a seniorifirst-year
graduate level course. However, in a number of areas those seeking
deeper understanding either for complex applications or through an
interest in research will have to seek out either one of the newer
texts or follow their interests through the references provided to other
sources.
REFERENCES
[l] E. Amaldi, “From finding maximum feasible subsystems of linear
systems to feedforward neural network design,” Sc.D. dissertation, Dept.
Math., ETH Lausanne, EPFL, 1994.
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 42, NO. 4 .JULY 1996
E. Baum and D. Haussler, “What size net gives valid generalization?,” in
D. Touretzky, Ed., Advances in Neural Information Processing Systems
I . San Mateo, CA: Morgan Kaufmann, 1989, pp. 81-90.
T. M. Cover, “Geometrical and statistical properties of systems of
linear inequalities with applications in pattern recognition,” IEEE Trans.
Electron. Comput., pp. 326-334, 1965.
M. Hassoun, Fundamentals ofArtificial Neural Networks. Cambridge,
MA: MIT Press, 1995.
S. Haykin, Neural Networks: A Comprehensive Foundation. New
York: Macmillan, 1994.
J. Hertz, A. Krogh, and R. Palmer, Introduction to the Theory of Neural
Computation. Redwood City, CA: Addison-Wesley, 1991.
J. Hopfield, “Neural networks and physical systems with emergent
collective computational abilities,” Proc. Nut. Acad. Sciences, vol. 79,
pp. 2554-2558, 1982.
R. McEliece, E. Posner, E. Rodemich, and S. Venkatcsh, “The capacity
of the Hopfield associative memory,” IEEE Trans. Inform Theory, vol.
IT-33, pp. 461482, 1987.
M. Moeller, “A scaled conjugate gradient algorithm for fast supervised
learning,” Neural Networks, vol. 6 , pp. 525-533, 1993.
N. Nilsson, The Mathematical Foundations of Learning Machines
(1965). San Mateo, CA: Morgan Kaufmann, 1990, reprint.
B. Ripley, Pattern Recognition and Neural Networks. Cambridge, UK:
Cambridge Univ. Press, 1996.
F. Rosenblatt, Principles of Neurodynamics: Perceptrons and the Theory
of Brain Mechanisms (a version of a 1961 report of the same title done
for Come11 Aeronautical Labs.). Washington, DC: Spartan Books,
1962.
D. E. Rumelhart and J. L. McClelland, Eds. Parallel Distributed
Processing. Cambridge MA: MIT Press, 1986.
K-Y. Siu, V. Roychowdhury, and T. Kailath, Discrete Neural Computa-
tiont A Theoretical Foundation. Englewood Cliffs, NJ: Prentice-Hall,
1995.
V. Vapnik, The Nature of Statistical Learning Theory. New York:
Springer-Verlag, 1995.
J. Zurada, Introduction to Artificial Neural Systems. St. Paul, MN: West
Publishing, 1992.
Terrence I. Fine (S’62-M’63-SM’81-F’82) received the Ph.D. degree in
applied physics from Harvard University, Cambridge, MA, in 1963.
After a post-doctoral appointment at Harvard and serving as a Miller
Fellow ut the University of California ut Berkeley, he joined the School of
Electrical Engineering at Cornell University, Ithaca, NY, in 1966. His research
interests are in the foundation of probability, particularly when probability is
not represented by the r e d s and in statistical aspects of the performance and
design oj neural networks.
Dr. Fine has been an Associate Editor f o r Detection and Estimation and f o r
book reviews of the IEEE TRANSACTIONS ON INFORMATION THEORY and is U Past-
President of the IEEE Information Theory Society (actually, the lust President
of the Information Theory Group). H e is also a founding member of the Board
of Directors of the Neural Information Processing Systems Foundution.