ELSEVIER BioSystems33 ( 1994) 69-73
Book Review
Genetic Programming: On the Programming of
Computers by Means of Natural Selection, John R.
Koza, A Bradford Book, MIT Press, Cambridge
MA, 1992, ISBN o-262-11170-5, xiv + 819pp.,
US%55.00
Problem-solving, function optimization, search
and learning are engineering problems that presup
pose the attainment of a goal. Genetic algorithms
(Holland, 1975; Goldberg, 1989), evolution strate-
gies (Rechenberg, 1973; Schwefel, 1981; B&k et al.,
1991), evolutionary programming (Fogel et al.,
1966; Fogel 1992), and other methods that are col-
lectively known as evolutionary computations, are
computational approaches for such engineering
tasks. While the question of whether or not natural
evolution is teleological remains unresolved, resear-
chers who use these algorithmic analogies to evo-
lution almost invariably do so for their apparent
goal-directed behavior. It is no wonder then that
Genetic Programming: On the Programming of Com-
puters by Means of Natural Selection is written large-
ly for an engineering audience.
This large hardcover volume describes a recent
addition to the set of evolutionary computations
and applies it to a diverse collection of engineering
problems. The technique, which Koza calls genetic
programming, is a straightforward conceptual rela-
tive of genetic algorithms. A task, be it an instance
of function optimization, problem-solving, or other
engineering problem, is represented as a fitness func-
tion, $S - %, where the value returned indicates
the proximity of the structure S to a task solution.
As in all evolutionary computations, a population
of structures is manipulated in parallel. Once all
structures in the population are rated by the fitness
function, these values are used to determine their
0303-2647/94/$07.00 0 1994 Elsevier Scicncc lrcland Ltd. All rights reserved
SSDI 0303-2647(93)01433-T
relative reproductive rates. A stochastic process
then creates the next population using the corn-
puted reproductive rates and special operators for
mutation and recombination of structures.
True to the technique’s name, the individuals of
the population in genetic programming are pro-
grams and thus are evaluated by executing them in
the task environment. Solving problems by evolv-
ing programs seems an obvious application for com-
putational implementations of evolution. However,
two qualities of traditional programming languages
discourage this line of experimentation. First, stan-
dard programming languages, such as C, Pascal,
and FORTRAN are very general, low-level and rely
heavily on correct syntax to disambiguate the in-
tended behavior. The standard manipulations of ge-
netic algorithms generate unsuitable results when
applied to such languages since straightforward ap-
plications of crossover are likely to construct pro-
grams that are syntactically incorrect. Also, a high
probability of creating non-viable offspring retards
the rate of evolutionary progress. Second, there is
no way to anticipate the size of an appropriate pro-
gram for a task. Thus, the fixed-length binary string
representation used in genetic algorithms can pre-
vent the discovery of a correct program simply be-
cause the solution is larger than the assumed
fixed-length.
Genetic programming alleviates these problems
by making two basic changes to the standard gene-
tic algorithm. First, each problem uses a language
explicitly designed for the task at hand which
reduces the expressiveness of the language as well
as the synactic constants. For instance, if the task
is to fit a function to a set of data points, the pro-
gramming language is designed with an appropri-
ate number of variables and suitable mathematical
70 Book review / BioSystenrs .t3 (1994)
operators. if, instead, the problem is to create a pro-
gram to control an artificial ant along a path of
food, Koza uses a different language for the
representation of programs. Essentially, each new
problem requires its own distinct programming lan-
guage. This presents an immediate problem because
the difficulty of solving a task is strongly affected
by the choice of programming language. It is rea-
sonable then to question to what extent the results
Koza shows are due to his technique or are artifacts
of his choice of programming language. 1 return to
this point below.
The second difference of genetic programming
from genetic algorithms is the representation of the
individual programs. Rather than a fixed-length
binary string, genetic programs are stored as com-
plex expressions in the given language using pre$
notation. Prefix notation denotes an expression
ordering where the operator is followed by the cor-
rect number of arguments. If, in turn, each argu-
ment is itself an expression, a recursive tree structure
develops with the operators as internal nodes of the
tree and the operands as the leaves, as shown in Fig.
1. Borrowing the syntactic convention of LISP, a
programming language most often associated with
artificial intelligence, Koza denotes complete sub-
trees in his evolved programs by offsetting them
with parentheses, as shown in the figure.
Program manipulation in genetic programming
is carried out by a specialized crossover operation
that recombines two parent genetic programs into
./@l
i\ /\
dl not d2
cl2 d3
(a)
Fig. I. (a) A boolean expressionin tree form. Operators are internal nodes while variables are leavesof the tree. (b) Prefix notation
of booiean expressionin (a) using parenthesesas expression delimiters.
69-73
syntactically valid offspring. Recombination simp-
ly consists of selecting a complete subtree from each
of the parent programs and swapping them. Because
all individuals in the population are represented as
recursive tree structures, simply removing a com-
plete subtree and replacing it with another auto-
matically preserves all syntactic constraints. Cramer
(1985) also used this recombination operator for the
manipulation of a similar representation.
As expression trees are recursive, they are of
variable depth and therefore of unlimited size.
Although Koza claims his programs take full ad-
vantage of this dynamic representation, the code
supplied in Appendix C of the book uses a specific
variable to limit the ultimate size of the evolving ex-
pression trees. Otherwise, the structures grow with-
out bound and quickly till the computer’s available
memory. Such a limitation is conceptually impor-
tant when trying to assess the differences between
genetic programming and standard genetic algo-
rithms.
Undoubtedly, this book’s most worthwhile fea-
ture is the multitude of solved problems that dem-
onstrate the viability of this technique. A reader
leafing through its contents might at first mistake
it for an engineering primer, describing classic pro-
blems from a number of fields including control,
planning, sequence induction, symbolic regression,
strategy induction, classification and cellular auto-
mata. In the end, no less than 50 problems are solv-
ed each with several experiments containing
or
(and
dl (and d 1 (not (or d2 d3)))
(or d2 dl))
(W
 Book review / BioSytems
numerous runs of genetic programming. Such rig-
orous experimentation is noteworthy.
While Koza provides impressive experimental val-
idation for his technique, he never generalizes any
theory from these numerous results. Instead, he
seems content to merely validate the technique over
and over again. To borrow an apt quote from Pir-
sig (1992) ‘Data without generalization is just
gossip’. Even a modicum of speculation to guide the
reader through the myriad of experiments would
help to place the results into perspective and pro-
vide insight into the natural biases of the technique.
As is, the book appears to offer a computational
panacea, which ultimately compromises the
technique.
Koza does provide comparisons between genetic
programming and other computational techniques,
including hill-climbing, neural networks, simulated
annealing and ID3, a classic symbolic learning
method. However, these comparisons are superficial
and concentrate only on the interface between a task
and the techniques. Nowhere are comparative per-
formance estimates shown between genetic pro-
gramming and any other method. As a result, Koza
builds an island for genetic programming separate
from all other problem-solving and optimization
techniques, a criticism that is equally valid for other
evolutionary computations. Understanding the ad-
vantages and disadvantages of this entire class of
computational techniques should be a collective
goal of this burgeoning field. Otherwise, all of these
methods are interesting only by virtue of their
resemblance to evolution, a feature that will be at-
tractive for a limited time. In spite of its wealth of
experimentation, this book brings the field no closer
to that goal.
The ease by which a problem is addressed with
genetic programming is strongly dependent on the
appropriateness of the programming language pro-
vided. Languages that contain primitives which rep-
resent concepts central to solving the problem
provide overwhelming advantages over languages
that are less problem specific. For instance, Koza
solves a classification problem where two sets of
points lie along intertwined spirals on the xy-plane.
A correct genetic program is expected to identify
which spiral a particular point belongs to given only
its coordinates. To solve the interlocking spiral
33 (1994) 69-73 71
problem, Koza defines a language that includes sine
and cosine as primitive functions. Given that the
equation to generate a spiral uses these functions,
this is a prudent addition. It is not surprising then
that the experiment evolved a correct program
within 36 generations. Problem-specific knowledge
of the origin of the two sets of points allowed for
the selection of a suitable language such that a cor-
rect program was a simple consequence. There is
nothing wrong with this, it is common practice when
demonstrating a new technique, but an engineer
using genetic programming is unlikely to be in
possession of equivalent knowledge.
Let us call such a situation, where the solution
to the problem is a simple consequence of the
representation, a level 0 problem’, meaning that
the solution is ‘the same level of difficulty’ as the
solution’s representation. Level 0 problems are the
norm experiment in evolutionary computations and
in most other problem-solving and machine learn-
ing methods. In all these methods, the solutions are
usually direct consequences of the provided repre-
sentation. Often the practitioners of these tech-
niques even stress the importance of selecting an ap-
propriate representation for a problem to ensure
success with the technique.
What would it mean if the solution to a problem
is not a simple consequence of the representation,
i.e. the problem is not a level 0 problem? Consider
the case of the interlocking spiral task described
above. It is easier to construct a solution for this
problem in a general mathematical language after
creating functions for sine and cosine rather than
attempting to create a solution without them. The
sine and cosine functions reduce the conceptual dif-
ficulty of creating the solution by raising the
abstractness of the available representational com-
ponents. In other words, sine and cosine provide
a semantically distinct level of representation that
is more conducive to evolving the solution for the
two spirals problem.
Let a level 1 problem signify that there is a single
intermediate level of representation definable above
the given representation that allows the solution to
‘Thisisdistinctfromthe concept of level in Fontana (1992)and
Fontam and BUSS (1993) although the Concepts are not COm-
pletely disjoint.
72 Book review / BioSystems 33 (1994)
be created more efficiently. In the case of writing
a program, a level 1 problem suggests that by first
defining a single level of functions, the program can
be written much more concisely. When the most ef-
ficient method to represent a solution is to first
create two levels of functions, i.e. some function is
most eniciently defined by first defining a different
function, the problem would be a level 2 problem.
To generalize, a level n problem would then require
n distinct levels of representation above that pro-
vided for the solution to be represented efficiently.
Each level is semantically distinct, in that it raises
the expressiveness of the provided representation
closer to the ultimate solution.
We should be wary of too many techniques that
only address level 0 tasks without a concerted at-
tack toward moving up this representational lad-
der, The key to moving beyond level 0 solutions is
to create methods that can identify the need for
building a representation up towards the solution
rather than trying to solve every problem in a sin-
gle semantic step. Moving up semantic levels pro-
vides the same benefit as describing DNA as a
collection of genes as opposed to an ordering of
atoms. Such progressions are imperative for open-
ended evolutionary computations to be tractable.
Koza offers a method for automatically evolving
functions in genetic programs with the intention of
doing exactly this. When using automatically defin-
ed functions (ADF), each population member con-
sists of both a main program and a collection of
functions. The parameters and language for each
function, as with the main program, are individually
defined. For instance, the programming language
for the main program would contain the names of
the first level of ADFs while the language for the
functions in first level of ADFs would contain the
names of the second level of ADFs and so on up
rhe various representational levels. As a population
member’s main program is evolved. its associated
functions are also manipulated using the same
recombination operator. Only the definitions of the
various ADFs and the main program are manipul-
ated; the number of distinct levels and their com-
position are static. Consequently. ADFs require
knowledge of how to define and organize the seman-
tic levels prior to solving the problem. As a result,
this technique only distributes the representational
69-73
and semantic load between the pre-defined levels
rather than automatically constructing an appropri-
ate semantic abstraction for the task. More infor-
mative work in such evolutionary progressions, with
fewer restrictions, has appeared in the artificial life
community (Fontana 1992; Ray 1992; Fontana and
Buss 1993; Angeline and Pollack 1994). Each of
these studies, like Koza, uses a programming lan-
guage variant as the base representation.
The longevity of Koza’s technique is strongly
linked with the fate of genetic algorithms. The con-
verse is probably true as well, especially given that
evolving a program is conceptually more appealing
than the manipulation of a fixed-length bit string.
This book serves as a line, although repetitive, in-
troduction and should be an inspiration to those
new to evolutionary computations. However, the
tacit promises of a generally applicable technique
are reminiscent of the inflated claims of expert sys-
tems from only a decade ago. Practitioners of this
new computational art should describe the limits of
these techniques accurately so that genetic pro-
gramming and other evolutionary computations
help engineers, rather than frustrate them.
References
Angeline,P.J. and Pollack, J.B., 1994, Coevolving high-level
representations.Artificial Life, 111. C. Langton (ed.)
(Addison-Wesley, Reading,MA)pp. 55-71.
BLk. T.. Hoffmeister. F. and Schwefel, H.-P., 1991. A survey
of evolutionstrategies,in: Proceedingsof the Fourth fnter-
national Conferenceon Genetic Algorithms, R.K. Belew and
L.B. Booker(eds.)(Morgan Kaufmann Publishers,San Mateo
CA) pp. 2-9.
Cramer, M.L.. 1985. A representationfor the adaptive genera-
tion of simple sequentialprograms,in: Proceedingsof ae-ln-
ternational Conference on Genetic Algorithms and their
Applications, J. Grefenstette (ed.) (Lawrence Erlbaum
Associates,Hillsdale NJ) pp. 183-187.
Fogel. D.B. 1992, Evolving Artificial Intelligence. Doctoral
Dissertation, University of California at San Diego.
Fogel. L.J., Owens, A.J. and Walsh, M.J., 1966. Artificial Intel-
ligence through Simulated Evolution (John Wiley & Sons,
New York).
Fontana. W.. 1992. Algorithmic chemistry. in: Artificial Life Il.
C. Langton. C. Taylor. J. Farmer and S. Rasmussen(eds.),
(Addison-Wesley, Reading MA) pp. 157-209.
Fontana. W. and Buss, L.W., 1993, The arrival of the fittest:
toward a theory of biological organization. Bull Math Biol
(in press).
 Book review / BioSystems 33 (1994) 69-73
Goldberg, D.E.. 1989, Genetic Algorithms in Search, Optimi-
zarion. and Machine Learning. (Addison-Wesley. Reading
MA).
Holland, J.H.. 1975, Adaptation in natural and artificial systems
(The University of Michigan Press. Ann Arbor MI).
Pirsig, R.M., 1992. Lila, an inquiry into morals (Bantam Books,
New York).
Ray, T.. 1992, An approach to the synthesis of life, in: Artificial
Life IL C. Langton, C. Taylor, J. Farmer and S. Rasmussen
(eds.) (Addison-Wesley, Reading MA) pp. 371408.
73
Rechenberg, I., 1973, Evolutionsstrategie, Optimierung
Technischer Systeme nach Prinzipien der Biologischen Evo-
lution (Frommann-Holtboog Verlag, Stuttgart).
Schwefel. H.-P., 1981, Numerical Optimization of Computer
Models (John Wiley & Sons, Chichester).
Peter J. Angeline
Laboratory for Artificial Intelligence Research,
Department of Computer and Information Science,
The Ohio State University,
Columbus, OH 43210, USA