Professional Documents
Culture Documents
http://mitpress.mit.edu/journals
To join an e-mail alert list and receive the latest news on our publications, please visit:
http://mitpress.mit.edu/e-mail
Evolving Neural Networks through
Augmenting Topologies
Abstract
An important question in neuroevolution is how to gain an advantage from evolving
neural network topologies along with weights. We present a method, NeuroEvolu-
tion of Augmenting Topologies (NEAT), which outperforms the best fixed-topology
method on a challenging benchmark reinforcement learning task. We claim that the
increased efficiency is due to (1) employing a principled method of crossover of differ-
ent topologies, (2) protecting structural innovation using speciation, and (3) incremen-
tally growing from minimal structure. We test this claim through a series of ablation
studies that demonstrate that each component is necessary to the system as a whole
and to each other. What results is significantly faster learning. NEAT is also an im-
portant contribution to GAs because it shows how it is possible for evolution to both
optimize and complexify solutions simultaneously, offering the possibility of evolving
increasingly complex solutions over generations, and strengthening the analogy with
biological evolution.
Keywords
Genetic algorithms, neural networks, neuroevolution, network topologies, speciation,
competing conventions.
1 Introduction
Neuroevolution (NE), the artificial evolution of neural networks using genetic algo-
rithms, has shown great promise in complex reinforcement learning tasks (Gomez and
Miikkulainen, 1999; Gruau et al., 1996; Moriarty and Miikkulainen, 1997; Potter et al.,
1995; Whitley et al., 1993). Neuroevolution searches through the space of behaviors for
a network that performs well at a given task. This approach to solving complex control
problems represents an alternative to statistical techniques that attempt to estimate the
utility of particular actions in particular states of the world (Kaelbling et al., 1996). NE is
a promising approach to solving reinforcement learning problems for several reasons.
Past studies have shown NE to be faster and more efficient than reinforcement learn-
ing methods such as Adaptive Heuristic Critic and Q-Learning on single pole balanc-
ing and robot arm control (Moriarty and Miikkulainen, 1996; Moriarty, 1997). Because
NE searches for a behavior instead of a value function, it is effective in problems with
continuous and high-dimensional state spaces. In addition, memory is easily repre-
sented through recurrent connections in neural networks, making NE a natural choice
for learning non-Markovian tasks (Gomez and Miikkulainen, 1999, 2002).
c
2002 by the Massachusetts Institute of Technology Evolutionary Computation 10(2): 99-127
K. O. Stanley and R. Miikkulainen
The NEAT method consists of solutions to each of these problems as will be de-
scribed below. The method is validated on pole balancing tasks, where NEAT per-
forms 25 times faster than Cellular Encoding and 5 times faster than ESP. The results
show that structure is a powerful resource in NE when appropriately utilized. NEAT
is unique because structures become increasingly more complex as they become more
optimal, strengthening the analogy between GAs and natural evolution.
2 Background
Many systems have been developed over the last decade that evolve both neural net-
work topologies and weights (Angeline et al., 1993; Braun and Weisbrod, 1993; Das-
gupta and McGregor, 1992; Fullmer and Miikkulainen, 1992; Gruau et al., 1996; Krish-
nan and Ciesielski, 1994; Lee and Kim, 1996; Mandischer, 1993; Maniezzo, 1994; Opitz
and Shavlik, 1997; Pujol and Poli, 1998; Yao and Liu, 1996; Zhang and Mühlenbein,
1993). These methods encompass a range of ideas about how Topology and Weight Evolv-
ing Artificial Neural Networks (TWEANNs) should be implemented. In this section, we
address some of the ideas and assumptions about the design of TWEANNs, and of-
fer solutions to some unsolved problems. Our goal is to find how a neuroevolution
method can use the evolution of topology to increase its efficiency.
2.1.3 Nonmating
Because crossover of networks with different topologies can frequently lead to a loss of
functionality, some researchers have given up on crossover altogether in what is called
Evolutionary Programming (Yao and Liu, 1996). Angeline et al. (1993) implemented
a system called GeNeralized Acquisition of Recurrent Links (GNARL), commenting that
“the prospect of evolving connectionist networks with crossover appears limited in
general.” Although GNARL uses a graph encoding, it is fundamentally different from
PDGP in that it sidesteps the issue of crossover entirely. GNARL demonstrates that a
TWEANN does not need crossover to work, leaving the problem of demonstrating the
advantages of crossover to other methods.
3 3
A B C C B A
1 2 1 2
[A,B,C]
x [C,B,A]
Crossovers: [A,B,A] [C,B,C]
(both are missing information)
Figure 1: The competing conventions problem. The two networks compute the same
exact function even though their hidden units appear in a different order and are repre-
sented by different chromosomes, making them incompatible for crossover. The figure
shows that the two single-point recombinations are both missing one of the 3 main
components of each solution. The depicted networks are only 2 of the 6 possible per-
mutations of hidden unit orderings.
We now turn to several specific problems with TWEANNs and address each in
turn.
this area.” Although some TWEANNs such as PDGP have attempted to address the
problem by assuming that subnetworks represent functional units that can be recom-
bined, different topologies may not be based on the same subnetworks at all, in which
case no meaningful combination of substructures exists.
The main intuition behind NEAT originates from the fundamental problem with
representing different structures: their representations will not necessarily match up.
Sometimes, the genomes can have different sizes. Other times, genes in the exact same
position on different chromosomes may be expressing completely different traits. In
addition, genes expressing the same trait may appear at different positions on different
chromosomes. How can these complications be resolved?
Nature faces a similar problem with gene alignment in sexual reproduction.
Genomes in nature are not of fixed-length either. Somewhere along the evolution from
single cells to more complex organisms, new genes were added to the genomes in a
process called gene amplification (Darnell and Doolittle, 1986; Watson et al., 1987). If
new genes could just randomly insert themselves in positions on the genome without
any indication of which gene is which, life probably would not have succeeded, be-
cause the competing conventions problem would decimate a huge chunk of offspring.
There needed to be some way to keep crossover orderly, so that the right genes could
be crossed with the right genes.
Nature’s solution utilizes homology: two genes are homologous if they are alleles
of the same trait. For example, in E. coli, in a process called synapsis, a special protein
called RecA goes through and lines up homologous genes between two genomes before
crossover occurs (Radding, 1982; Sigal and Alberts, 1972). Actual homology between
neural networks cannot be easily ascertained by direct structural analysis (hence, the
competing conventions problem). The main insight in NEAT is that the historical origin
of two genes is direct evidence of homology if the genes share the same origin. Thus,
NEAT performs artificial synapsis based on historical markings, allowing it to add new
structure without losing track of which gene is which over the course of a simulation.
have a chance to optimize their structures before having to compete with the popula-
tion at large.
Speciation, also known as niching, has been studied in GAs, but is not usually ap-
plied to neuroevolution. Speciation is most commonly applied to multimodal function
optimization (Mahfoud, 1995), where a function has multiple optima, and a GA with
several species is used to find those optima. Speciation has also been applied in the
cooperative coevolution of modular systems of multiple solutions (Darwen and Yao,
1996; Potter and De Jong, 1995).
Speciation requires a compatibility function to tell whether two genomes should
be in the same species or not. It is difficult to formulate such a compatibility function
between networks of different topologies, which may be the reason why speciation
has not been brought into TWEANNs. The competing conventions problem makes
measuring compatibility particularly problematic because networks that compute the
same function can appear very different.
However, because NEAT has a solution to the competing conventions problem,
using historical information about genes, the population in NEAT can easily be spe-
ciated. We use explicit fitness sharing, which forces individuals with similar genomes
to share their fitness payoff (Goldberg and Richardson, 1987). The original implicit
version of fitness sharing introduced by Holland (1975) grouped individuals by perfor-
mance similarity rather than genetic similarity. The explicit version is appropriate for
TWEANNs because it allows grouping networks based on topology and weight con-
figurations. The result of sharing fitness is that the number of networks that can exist
in the population on a single fitness peak is limited by the size of the peak. Therefore,
the population divides into a number of species, each on a different peak, without the
threat of any one species taking over. Explicit fitness sharing is well-suited for NEAT,
because similarity can easily be measured based on the historical information in the
genes. Thus, innovations in NEAT are protected in their own species.
Genome (Genotype)
Node Node 1 Node 2 Node 3 Node 4 Node 5
Genes Sensor Sensor Sensor Output Hidden
Connect. In 1 In 2 In 3 In 2 In 5 In 1 In 4
Out 4 Out 4 Out 4 Out 5 Out 4 Out 5 Out 5
Genes Weight 0.7 Weight−0.5 Weight 0.5 Weight 0.2 Weight 0.4 Weight 0.6 Weight 0.6
Enabled DISABLED Enabled Enabled Enabled Enabled Enabled
Innov 1 Innov 2 Innov 3 Innov 4 Innov 5 Innov 6 Innov 11
Network (Phenotype) 4
5
1 2 3
fitness function in this way can encourage smaller networks, it is difficult to know
how large the penalty should be for any particular network size, particularly because
different problems may have significantly different topological requirements. Altering
the fitness function is ad hoc and may cause evolution to perform differently than the
designer of the original unmodified fitness function intended.
An alternative solution is for the neuroevolution method itself to tend towards
minimality. If the population begins with no hidden nodes and grows structure only
as it benefits the solution, there is no need for ad hoc fitness modification to minimize
networks. Therefore, starting out with a minimal population and growing structure
from there is a design principle in NEAT.
By starting out minimally, NEAT ensures that the system searches for the solu-
tion in the lowest-dimensional weight space possible over the course of all generations.
Thus, the goal is not to minimize only the final product, but all intermediate networks
along the way as well. This idea is they key to gaining an advantage from the evo-
lution of topology: it allows us to minimize the search space, resulting in dramatic
performance gains. One reason current TWEANNS do not start out minimally is that
without topological diversity present in the initial population, topological innovations
would not survive. The problem of protecting innovation is not addressed by these
methods, so networks with major structural additions are likely not to reproduce. Thus,
speciating the population enables starting minimally in NEAT.
1 2 3 4 5 6 1 2 3 4 5 6 7
1−>4 2−>4 3−>4 2−>5 5−>4 1−>5 1−>4 2−>4 3−>4 2−>5 5−>4 1−>5 3−>5
DIS DIS
Mutate Add Connection
4 4
5 5
1 2 3 1 2 3
1 2 3 4 5 6 1 2 3 4 5 6 8 9
1−>4 2−>4 3−>4 2−>5 5−>4 1−>5 1−>4 2−>4 3−>4 2−>5 5−>4 1−>5 3−>6 6−>4
DIS DIS DIS
Mutate Add Node
4 4
5 5
6
1 2 3 1 2 3
Figure 3: The two types of structural mutation in NEAT. Both types, adding a connec-
tion and adding a node, are illustrated with the connection genes of a network shown
above their phenotypes. The top number in each genome is the innovation number of
that gene. The innovation numbers are historical markers that identify the original his-
torical ancestor of each gene. New genes are assigned new increasingly higher num-
bers. In adding a connection, a single new connection gene is added to the end of the
genome and given the next available innovation number. In adding a new node, the
connection gene being split is disabled, and two new connection genes are added to the
end the genome. The new node is between the two new connections. A new node gene
(not depicted) representing this new node is added to the genome as well.
the genetic encoding used in NEAT and continue by describing the components that
specifically address each of the three problems of TWEANNs.
split and the new node placed where the old connection used to be. The old connection
is disabled and two new connections are added to the genome. The new connection
leading into the new node receives a weight of 1, and the new connection leading out
receives the same weight as the old connection. This method of adding nodes was cho-
sen in order to minimize the initial effect of the mutation. The new nonlinearity in the
connection changes the function slightly, but new nodes can be immediately integrated
into the network, as opposed to adding extraneous structure that would have to be
evolved into the network later. This way, because of speciation, the network will have
time to optimize and make use of its new structure.
Through mutation, the genomes in NEAT will gradually get larger. Genomes of
varying sizes will result, sometimes with different connections at the same positions.
The most complex form of the competing conventions problem, with numerous differ-
ing topologies and weight combinations, is an inevitable result of allowing genomes to
grow unbounded. How can NE cross over differently sized genomes in a sensible way?
The next section explains how NEAT addresses this problem.
Parent1 Parent2
1 2 3 4 5 8 1 2 3 4 5 6 7 9 10
1−>4 2−>4 3−>4 2−>5 5−>4 1−>5 1−>4 2−>4 3−>4 2−>5 5−>4 5−>6
6−>4 3−>5 1−>6
DISAB DISAB DISAB
4 4
5 6
5
1 2 3
1 2 3
disjoint
1 2 3 4 5 8
Parent1
9 10
1 2 3 4 5 6 7
Parent2 1−>4 2−>4 3−>4 2−>5 5−>4 5−>6
DISAB DISAB
excess excess
disjoint disjoint
1 2 3 4 5 6 7 8 9 10
Offspring
1−>4 2−>4 3−>4 2−>5 5−>4 5−>6 6−>4 1−>5 3−>5 1−>6
DISAB DISAB
1 2 3
protected in a new niche where they have time to optimize their structure through
competition within the niche. The idea is to divide the population into species such
that similar topologies are in the same species. This task appears to be a topology
matching problem. However, it again turns out that historical markings offer an effi-
cient solution.
The number of excess and disjoint genes between a pair of genomes is a natural
measure of their compatibility distance. The more disjoint two genomes are, the less
evolutionary history they share, and thus the less compatible they are. Therefore, we
can measure the compatibility distance δ of different structures in NEAT as a simple lin-
ear combination of the number of excess E and disjoint D genes, as well as the average
weight differences of matching genes W , including disabled genes:
c1 E c2 D
δ= + + c3 · W . (1)
N N
The coefficients c1 , c2 , and c3 allow us to adjust the importance of the three factors, and
the factor N , the number of genes in the larger genome, normalizes for genome size (N
can be set to 1 if both genomes are small, i.e., consist of fewer than 20 genes).
The distance measure δ allows us to speciate using a compatibility threshold δt .
An ordered list of species is maintained. In each generation, genomes are sequentially
placed into species. Each existing species is represented by a random genome inside
the species from the previous generation. A given genome g in the current generation is
placed in the first species in which g is compatible with the representative genome of
that species. This way, species do not overlap.1 If g is not compatible with any existing
species, a new species is created with g as its representative.
As the reproduction mechanism for NEAT, we use explicit fitness sharing (Goldberg
and Richardson, 1987), where organisms in the same species must share the fitness
of their niche. Thus, a species cannot afford to become too big even if many of its
organisms perform well. Therefore, any one species is unlikely to take over the entire
population, which is crucial for speciated evolution to work. The adjusted fitness fi0 for
organism i is calculated according to its distance δ from every other organism j in the
population:
fi
fi0 = Pn . (2)
j=1 sh(δ(i, j))
The sharing function sh is set to 0 when distance δ(i,
Pnj) is above the threshold δt ;
otherwise, sh(δ(i, j)) is set to 1 (Spears, 1995). Thus, j=1 sh(δ(i, j)) reduces to the
number of organisms in the same species as organism i. This reduction is natural since
species are already clustered by compatibility using the threshold δt . Every species is
assigned a potentially different number of offspring in proportion to the sum of ad-
justed fitnesses fi0 of its member organisms. Species then reproduce by first eliminating
the lowest performing members from the population. The entire population is then
replaced by the offspring of the remaining organisms in each species.2
The net desired effect of speciating the population is to protect topological inno-
vation. The final goal of the system, then, is to perform the search for a solution as
efficiently as possible. This goal is achieved through minimizing the dimensionality of
the search space.
1 It is also possible to determine the compatibility of a genome g with a species s by using the average
compatibility of g with every genome in a species s, but in practice, only comparing to the first genome in s
is sufficient and takes constant time.
2 In rare cases when the fitness of the entire population does not improve for more than 20 generations,
only the top two species are allowed to reproduce, refocusing the search into the most promising spaces.
4 Performance Evaluations
We evaluate the system’s performance in order to answer two questions: (1) Can NEAT
evolve the necessary structures? (2) Can NEAT find solutions more efficiently than
other neuroevolution systems? The first question establishes that topology building
indeed happens in NEAT in a reliable way, meaning that NEAT will grow new structure
to cope with problems that require it. For this reason, NEAT is applied to the problem
of building an XOR network. Although this task is simple, it requires growing hidden
units, and therefore serves as a simple test for the method.
The second question is answered in the course of successively more difficult pole
balancing tasks, where the objective is to balance two poles attached to a cart by moving
the cart in appropriate directions to keep the pole from falling. Pole balancing is a good
benchmark task because there are many different systems available for comparison.
The most difficult problem of balancing two poles without velocity information, a non-
Markovian task, provides very strong evidence that evolving augmenting topologies
is not only interesting for its capacity to find structures, but is also efficient in difficult
control tasks.
4.1 Parameter Settings
The same experimental settings are used in all experiments; they were not tuned specif-
ically for any particular problem. The one exception is the hardest pole balancing prob-
lem (Double pole, no velocities, or DPNV) where a larger population size was used to
match those of other systems in this task. Because some of NEAT’s system parameters
are sensitive to population size, we altered them accordingly.
All experiments except DPNV, which had a population of 1,000, used a population
of 150 NEAT networks. The coefficients for measuring compatibility were c1 = 1.0,
c2 = 1.0, and c3 = 0.4. With DPNV, c3 was increased to 3.0 in order to allow for finer
distinctions between species based on weight differences (the larger population has
room for more species). In all experiments, δt = 3.0, except in DPNV where it was 4.0,
to make room for the larger weight significance coefficient c3 . If the maximum fitness of
a species did not improve in 15 generations, the networks in the stagnant species were
not allowed to reproduce. The champion of each species with more than five networks
was copied into the next generation unchanged. There was an 80% chance of a genome
having its connection weights mutated, in which case each weight had a 90% chance of
being uniformly perturbed and a 10% chance of being assigned a new random value.
(The system is tolerant to frequent mutations because of the protection speciation pro-
vides.) There was a 75% chance that an inherited gene was disabled if it was disabled
in either parent. In each generation, 25% of offspring resulted from mutation without
crossover. The interspecies mating rate was 0.001. In smaller populations, the proba-
bility of adding a new node was 0.03 and the probability of a new link mutation was
0.05. In the larger population, the probability of adding a new link was 0.3, because a
larger population can tolerate a larger number of prospective species and greater topo-
1
logical diversity. We used a modified sigmoidal transfer function, ϕ(x) = 1+e−4.9x , at
all nodes. The steepened sigmoid allows more fine tuning at extreme activations. It
is optimized to be close to linear during its steepest ascent between activations −0.5
and 0.5. These parameter values were found experimentally: links need to be added
significantly more often than nodes, and an average weight difference of 3.0 is about as
significant as one disjoint or excess gene. Performance is robust to moderate variations
in these values.
Out Out
2 3 2 3
Bias Bias
Phenotype of all genomes in initial population Phenotype of smallest possible solution
(No hidden nodes)
(a) (b)
Figure 5: Initial phenotype and optimal XOR. Figure (a) shows the phenotype given
to the entire initial population. Notice that there are no hidden nodes. In NEAT, a
bias is a node that can connect to any node other than inputs. Figure (b) shows an
optimal solution with only 1 hidden node. (A network without hidden nodes cannot
compute XOR.) The bias connections are not always needed depending on the solution;
All other connections are necessary. The optimal (1 hidden node) solution was found in
22 of 100 runs. The average solution had 2.35 hidden nodes with a standard deviation
of 1.11 nodes.
ficult in this domain because the methodologies vary widely across experiments (Das-
gupta and McGregor, 1992; Pujol and Poli, 1998; Yao and Shi, 1995; Zhang and
Mühlenbein, 1993). Also, XOR is simple and artificial and does not challenge modern
methods in the way real-world problems do. Let us next turn to benchmark problems
where more substantial performance comparisons are possible.
NE methods for two reasons: (1) The focus is on developing and demonstrating bet-
ter performance on evolving neural networks and (2) NE methods in this comparison
have outperformed reinforcement learning methods in prior comparisons on the pole
balancing task (Moriarty and Miikkulainen, 1996). Thus, the question here is whether
evolving structure can lead to greater NE performance.
4.3.1 Pole Balancing Comparisons
We set up the pole balancing experiments as described by Wieland (1991) and Gomez
and Miikkulainen (1999). The Runge-Kutta fourth-order method was used to imple-
ment the dynamics of the system, with a step size of 0.01s. All state variables were
scaled to [−1.0, 1.0] before being fed to the network. Networks output a force every
0.02 seconds between [−10, 10]N . The poles were 0.1m and 1.0m long. The initial po-
sition of the long pole was 1◦ and the short pole was upright; the track was 4.8 meters
long.
Two versions of the double pole balancing task are used: one with velocity inputs
included and another without velocity information. The first task is Markovian and
allows comparing to many different systems. Taking away velocity information makes
the task more difficult because the network must estimate an internal state in lieu of
velocity, which requires recurrent connections.
On the double pole balancing with velocity (DPV) problem, NEAT is compared
to published results from four other NE systems. The first two represent standard
population-based approaches. Saravanan and Fogel (1995) used Evolutionary Pro-
gramming, which relies entirely on mutation of connection weights, while Wieland
(1991) used both mating and mutation. The second two systems, SANE (Moriarty and
Miikkulainen, 1996) and ESP (Gomez and Miikkulainen, 1999), evolved populations of
neurons and a population of network blueprints that specifies how to build networks
from the neurons that are assembled into fixed-topology networks for evaluation. The
topologies are fixed because the individual neurons are always placed into predesig-
nated slots in the neural networks they compose. SANE maintains a single population
of neurons. ESP improves over SANE by maintaining a separate population for each
hidden neuron position in the complete network. To our knowledge, the results of ESP
are the best achieved so far in this task.
On the double pole balancing without velocity problem (DPNV), NEAT is com-
pared to the only two systems that have been demonstrated able to solve the task: CE
and ESP. The success of CE was first attributed to its ability to evolve structures. How-
ever, ESP, a fixed-topology NE system, was able to complete the task five times faster
simply by restarting with a random number of hidden nodes whenever it got stuck.
Our experiments will attempt to show that evolution of structure can lead to better
performance if done right.
4.3.2 Double Pole Balancing with Velocities
The criteria for success on this task was keeping both poles balanced for 100,000 time
steps (30 minutes of simulated time). A pole was considered balanced between -36 and
36 degrees from vertical. Fitness on this task was measured as the number of time steps
that both poles remained balanced.
Table 1 shows that NEAT takes the fewest evaluations to complete this task, al-
though the difference between NEAT and ESP is not statistically significant. The fixed-
topology NE systems evolved networks with 10 hidden nodes, while NEAT’s solutions
always used between 0 and 4 hidden nodes. Thus, it is clear that NEAT’s minimization
Table 1: Double pole balancing with velocity information (DPV). Evolutionary pro-
gramming results were obtained by Saravanan and Fogel (1995). Conventional neu-
roevolution data was reported by Wieland (1991). SANE and ESP results were reported
by Gomez and Miikkulainen (1999). In addition, Gruau et al. (1996) reported 34,000
evaluations in this task; however, their results are not directly comparable because they
used a different fitness function (Equations 3 and 4). NEAT results are averaged over
120 experiments. All other results are averages over 50 runs. The standard deviation
for the NEAT evaluations is 2,704 evaluations. Although standard deviations for other
methods were not reported, if we assume similar variances, all differences are statisti-
cally significant (p < 0.001), except that between NEAT and ESP.
f1 = t/1000, (3)
(
0 if t < 100,
f2 = Pt 0.75
otherwise. (4)
(|xi |+|ẋi |+|θ1i |+|θ̇1i |)
i=t−100
where t is the number of time steps the poles remain balanced during the 1,000 total
time steps. The denominator in (4) represents the sum of offsets from center rest of
the cart and the long pole. It is computed by summing the absolute value of the state
variables representing the cart and long pole positions and velocities. Thus, by mini-
mizing these offsets (damping oscillations), the system can maximize fitness. Because
of this fitness function, swinging the poles is penalized, forcing the system to internally
compute the hidden state variables.
Under Gruau et al.’s criteria for a solution, the champion of each generation is
tested on generalization to make sure it is robust. This test takes a lot more time than
the fitness test, which is why it is applied only to the champion. In addition to balancing
both poles for 100,000 time steps, the winning controller must balance both poles from
625 different initial states, each for 1,000 times steps. The number of successes is called
the generalization performance of the solution. In order to count as a solution, a network
needs to generalize to at least 200 of the 625 initial states. Each start state is chosen by
giving each state variable (i.e., x, ẋ, θ1 , and θ̇1 ) each of the values 0.05, 0.25, 0.5, 0.75,
0.95 scaled to the range of the input variable (54 = 625). At each generation, NEAT
performs the generalization test on the champion of the highest-performing species
that improved since the last generation.
Table 2 shows that NEAT is the fastest system on this challenging task. NEAT
takes 25 times fewer evaluations than Gruau’s original benchmark, showing that the
way in which structure is evolved has significant impact on performance. NEAT is also
5 times faster than ESP, showing that structure evolution can indeed perform better
than evolution of fixed topologies. There was no significant difference in the ability of
any of the 3 methods to generalize.
Why is NEAT so much faster than ESP on the more difficult task when there was
not much difference in the easier task? The reason is that in the task without veloc-
ities, ESP needed to restart an average of 4.06 times per solution while NEAT never
needed to restart. If restarts are factored out, the systems perform at similar rates. The
best characterization of the difference is that NEAT is more reliable at avoiding decep-
tion. NEAT evolves many different structures simultaneously in different species, each
representing a space of different dimensionality. Thus, NEAT is always trying many
different ways to solve the problem at once, so it is less likely to get stuck.
The experimental results demonstrate both that NEAT can evolve structure when
necessary, and that NEAT gains a significant performance advantage from doing so.
We now turn to understanding how the system works, and whether it indeed solves
the three problems with evolving a population of diverse topologies raised in the intro-
duction.
5 Analysis of NEAT
We have argued that NEAT’s performance is due to historical markings, speciation,
and incremental growth from minimal structure. In order to verify the contribution
of each component, we performed a series of ablations. In addition, we introduce a
new species visualization technique in order to better understand the dynamics of the
system.
Ablations are meant to establish that each component of NEAT is necessary for
its performance. For example, it is possible that growth from minimal structure is not
really important; maybe the rest of the system, speciation and historical markings, is
sufficient for NEAT’s optimal performance. This hypothesis will be checked by ablat-
ing both growth and starting from minimal structure from the system. On the other
Table 3: NEAT ablations summary. The table compares the average number of evalu-
ations for a solution in the double pole balancing with velocities task. Each ablation
leads to a weaker algorithm, showing that each component is necessary.
hand, perhaps the situation is the opposite, and speciation buys nothing: protecting in-
novation might not be as important as we have argued. This hypothesis will be checked
by ablating speciation from the system. Finally, we claimed that NEAT is able to make
use of crossover even though genomes in NEAT have different sizes. This point is
more controversial than it might seem. For example, Angeline et al. (1993) claimed that
crossover in TWEANNs does more harm than good. We will check this hypothesis by
ablating crossover from the system.
The reason why we do not ablate historical markings directly is that without his-
torical markings the system would be a conventional NE system. Historical markings
are the basis of every function in NEAT: Speciation uses a compatibility operator that is
based on historical markings, and crossover would not be possible without them. All
other system components can be ablated systematically.
Starting minimally
Historical Marking
Growth Speciation
hidden units like the fixed topology methods in this task (Saravanan and Fogel, 1995;
Wieland, 1991). Without growth, NEAT was still able to speciate, but only based on
weight differences. Given 1,000 generations to find a solution, the ablated system could
only find a solution 20% of the time! When it did find a solution, it took 8.5 times more
evaluations than full NEAT. Clearly, speciation and historical markings alone do not
account for full NEAT’s performance.
best-performing topology has about 5 hidden nodes. Thus, the population tends to
converge to a relatively high-dimensional search space, even though the smaller net-
works in the initial population would have optimized faster. The smaller networks just
do not get a chance because being small offers no immediate advantage in the initially
random weight space. Of course, once in a while small networks are found that per-
form well, allowing a solution to be found more quickly. Whether or not such networks
are found early on accounts for the large standard deviation of 41,704 evaluations.
The result shows that growth without speciation is not sufficient to account for
NEAT’s performance. For growth to succeed, it requires speciation, because speciation
gives different structures a chance to optimize in their own niches.
Figure 7: Visualizing speciation during a run of the double pole balancing with veloc-
ity information task. Two species begin to close in on a solution soon after the 20th
generation. Around the same time, some of the oldest species become extinct.
position and its already established method for comparing short pole velocity to long
pole angle. This innovation was enough to solve the problem within one generation
of additional weight mutations. In the final generation, the winning species was 11
generations old and included 38 neural networks out of the population of 150.
Most of the species that did not come close to a solution survived the run even
though they fell significantly behind around the 21st generation. This observation is
important, because it visually demonstrates that innovation is indeed being protected.
The winning species does not take over the entire population.
Ablation studies confirm the interdependence of all of NEAT’s components, and
the speciation visualization offers a means of visualizing the dynamics of the system.
We now turn to a discussion of the advantages and shortcomings of the method and its
future potential.
Figure 8: A NEAT solution to the DPNV problem. This clever solution works by taking
the derivative of the difference in pole angles. Using the recurrent connection to itself,
the single hidden node determines whether the poles are falling away or towards each
other. This solution allows controlling the system without computing the velocities of
each pole separately. Without evolving structure, it would be difficult to discover such
subtle and compact solutions. Starting minimally makes discovering such compact
solutions more likely.
output node of all solution networks. In 90% of solutions, this recurrent connection was
excitatory and served to maintain continuity in the actions. Upon this general theme,
solutions included varying degrees of structural elaborations utilizing from zero to two
hidden nodes. For example, the solution shown in Figure 8 uses one hidden node to
determine whether the poles are moving towards or away from one another. The basic
underlying structure is the same across all these solutions and suggests that once a
proper minimal foundation is discovered, a variety of paths open up to a solution.
In order to minimize structure throughout evolution, NEAT incrementally elabo-
rates structure in a stochastic manner from a minimal starting point. Because of spe-
ciation, useful elaborations survive even if they are initially detrimental. Thus, NEAT
strengthens the analogy between GAs and natural evolution by not only performing
the optimizing function of evolution, but also a complexifying function, allowing solu-
tions to become incrementally more complex at the same time as they become more
optimal.
It is this complexifying function that makes NEAT unique among GAs. Although
GAs have been proposed where bit string chromosomes can increase in length indef-
initely (Harvey, 1993), NEAT goes beyond a gradual uniform growth in genome size.
While Harvey’s method increases the size of a chromosome incrementally across the
entire population, NEAT simultaneously searches over different landscapes, all com-
plexifying in different ways.
Such speciated search of incrementally increasing complexity offers a possibly
powerful new approach to the problem of competitive coevolution. In competitive co-
evolution, increasingly sophisticated strategies are evolved by allowing networks in a
population to compete against each other. The hope is that an “arms race” will force the
opponents to continually evolve strategies better than the strategies of other networks
in the population. This method is useful because it can produce high-level strategies
and tactics without the need for an expert player to teach the system. Ideally, strategies
should become more sophisticated as evolution progresses. However, evolution tends
to find the simplest solutions that can win, meaning that strategies oscillate between
different idiosyncratic yet uninteresting variations (Darwen, 1996; Rosin and Belew,
1997).
We hypothesize that once competitive coevolution converges onto a dominant
strategy, it cannot be improved upon, because it takes the entire set of connection
weight values to represent the strategy. Altering the weights means altering the strat-
egy, rather than building upon and complexifying the strategy. Thus, if a new strategy
is to take hold, it must win by being different than the previous dominant strategy,
rather than by being more sophisticated. In contrast, NEAT can evolve increasingly more
sophisticated strategies continually, because as soon as the population converges on a
new dominant strategy, new connections and new nodes can be added to the current
strategy. New structure means new expressive space for elaborating on the existing strat-
egy, rather than replacing it. Thus, this approach allows continual coevolution, i.e., non-
convergent innovation on dominant strategies. In addition, because different strategies
in different species are protected, there will be multiple dominant and continually more
complex strategies.
In addition to continual coevolution, the evolution of structure should allow the
integration of separate expert neural networks. For example, suppose one neural net-
work can “kick” a ball towards the goal from any position on a field, and another net-
work can dribble the ball around the field without losing control. Neither of these two
networks alone can play soccer. However, if we could somehow combine their exper-
tise into one, perhaps we could get a soccer player out. Combining these controllers is
not a simple matter of processing both their outputs. Just because a robot can dribble
and shoot does not mean it knows where to dribble or when to shoot. Both shooting and
dribbling affect each other as well. Where you dribble affects how easy your shot is,
and shooting forces a robot to stop dribbling. In order to optimally combine the two
skills, the hidden nodes of the two networks must share information so that the new
combined expert can make intelligent decisions and combine the two skills effectively.
We hypothesize that NEAT has the capability of searching for the right interconnec-
tions between two distinct networks to create an integrated supernetwork that takes
advantage of the expertise of both its component networks.
Finally, we would like to establish a characterization of what NEAT is best suited
for. The experimental results show the difference between ESP and NEAT is signifi-
cantly higher on the hardest pole balancing task. This result implies that evolving di-
verse topologies is particularly suited for problems where other methods are likely to
get stuck. Such problems may be deceptive, meaning local optima have large basins of
attraction compared to global optima, and the global optima are significantly different
from the local optima (Goldberg, 1989). Because NEAT can always add more structure,
it is not necessarily trapped even if the current weights of a networks represent a local
optimum in fitness space. By adding additional structure, NEAT adds new dimensions
to weight space, thereby opening up potential new avenues for escape. We plan to test
NEAT on problems with varying fitness landscapes to get a better idea of the kinds of
problems the method tackles best.
7 Conclusion
The main conclusion is that NEAT is a powerful method for artificially evolving neural
networks. NEAT demonstrates that evolving topology along with weights can be made
a major advantage. Experimental comparisons verify that such evolution is several
times more efficient than the neuroevolution methods so far. Ablation studies show
that historical markings, protection of innovation through speciation, and incremental
growth from minimal structure all work together to produce a system that is capable of
evolving solutions of minimal complexity. NEAT strengthens the analogy between GAs
and natural evolution by both optimizing and complexifying solutions simultaneously.
We believe that the capacity to complexify solutions over the course of evolution offers
the possibility of continual competitive coevolution and evolution of combinations of
experts in the future.
Acknowledgments
This research was supported in part by the National Science Foundation under grant
IIS-0083776 and by the Texas Higher Education Coordinating Board under grant ARP-
003658-476-2001. Thanks to Faustino Gomez for providing pole balancing code and
Michael Shao for pointers to the biology literature.
References
Anderson, C. W. (1989). Learning to control an inverted pendulum using neural networks. IEEE
Control Systems Magazine, 9:31–37.
Angeline, P. J., Saunders, G. M., and Pollack, J. B. (1993). An evolutionary algorithm that con-
structs recurrent neural networks. IEEE Transactions on Neural Networks, 5:54–65.
Branke, J. (1995). Evolutionary algorithms for neural network design and training. In Alander,
J. T., editor, Proceedings First Nordic Workshop on Genetic Algorithms and their Applications,
pages 145–163, University of Vaasa Press, Vaasa, Finland.
Braun, H. and Weisbrod, J. (1993). Evolving feedforward neural networks. In Albrecht, R. F.,
Reeves, C. R., and Steele, N. C., editors, Proceedings of ANNGA93, International Conference on
Artificial Neural Networks and Genetic Algorithms, pages 25–32, Springer-Verlag, Innsbruck.
Chen, D. et al. (1993). Constructive learning of recurrent neural networks. In Petsche, T., Judd, S.,
and Hanson, S., editors, Computational Learning Theory and Natural Learning Systems III. MIT
Press, Cambridge, Massachusetts.
Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of
Control, Signals, and Systems, 2(4):303–314.
Darnell, J. E. and Doolittle, W. F. (1986). Speculations on the early course of evolution. Proceedings
of the National Academy of Sciences, USA, 83:1271–1275.
Darwen, P. J. (1996). Co-Evolutionary Learning by Automatic Modularisation with Speciation. Ph.D.
thesis, School of Computer Science, University College, University of New South Wales,
Sydney, Australia.
Darwen, P. and Yao, X. (1996). Automatic modularization by speciation. In Proceedings of the 1996
IEEE International Conference on Evolutionary Computation (ICEC ’96), pages 88–93, Nagoya,
IEEE Press, Piscataway, New Jersey.
Dasgupta, D. and McGregor, D. (1992). Designing application-specific neural networks using
the structured genetic algorithm. In Whitley, D. and Schaffer, J. D., editors, Proceedings of
the International Conference on Combinations of Genetic Algorithms and Neural Networks, pages
87–96, IEEE Press, Piscataway, New Jersey.
Fullmer, B. and Miikkulainen, R. (1992). Using marker-based genetic encoding of neural net-
works to evolve finite-state behaviour. In Varela, F. J. and Bourgine, P., editors, Toward a
Practice of Autonomous Systems: Proceedings of the First European Conference on Artificial Life,
pages 255–262, MIT Press, Cambridge, Massachusetts.
Goldberg, D. E. (1989). Genetic Algorithms in Search, Optimization and Machine Learning. Addison-
Wesley, Reading, Massachusetts.
Goldberg, D. E. and Richardson, J. (1987). Genetic algorithms with sharing for multimodal func-
tion optimization. In Grefenstette, J. J., editor, Proceedings of the Second International Confer-
ence on Genetic Algorithms, pages 148–154, Morgan Kaufmann, San Francisco, California.
Gomez, F. and Miikkulainen, R. (1997). Incremental evolution of complex general behavior. Adap-
tive Behavior, 5:317–342.
Gomez, F. and Miikkulainen, R. (1999). Solving non-Markovian control tasks with neuroevolu-
tion. In Dean, T., editor, Proceedings of the Sixteenth International Joint Conference on Artificial
Intelligence, pages 1356–1361, Morgan Kaufmann, San Francisco, California.
Gomez, F. and Miikkulainen, R. (2002). Learning robust nonlinear control with neuroevolution.
Technical Report AI02-292, Department of Computer Sciences, The University of Texas at
Austin, Austin, Texas.
Gruau, F. (1993). Genetic synthesis of modular neural networks. In Forrest, S., editor, Proceedings
of the Fifth International Conference on Genetic Algorithms, pages 318–325, Morgan Kaufmann,
San Francisco, California.
Gruau, F., Whitley, D., and Pyeatt, L. (1996). A comparison between cellular encoding and di-
rect encoding for genetic neural networks. In Koza, J. R. et al., editors, Genetic Programming
1996: Proceedings of the First Annual Conference, pages 81–89, MIT Press, Cambridge, Mas-
sachusetts.
Harvey, I. (1993). The Artificial Evolution of Adaptive Behavior Ph.D. thesis, School of Cognitive and
Computing Sciences, University of Sussex, Sussex, UK.
Holland, J. H. (1975). Adaptation in Natural and Artificial Systems: An Introductory Analysis with
Applications to Biology, Control and Artificial Intelligence. University of Michigan Press, Ann
Arbor, Michigan.
Kaelbling, L. P., Littman, M., and Moore, A. W. (1996). Reinforcement learning: A survey. Journal
of Artificial Intelligence, 4:237–285.
Krishnan, R. and Ciesielski, V. B. (1994). Delta-gann: A new approach to training neural net-
works using genetic algorithms. In Tsoi, A. C. and Downs, T., editors, Proceedings of the Aus-
tralian Conference on Neural Networks, pages 194–197, University of Queensland, Brisbane,
Australia.
Lee, C.-H. and Kim, J.-H. (1996). Evolutionary ordered neural network with a linked-list encod-
ing scheme. In Proceedings of the 1996 IEEE International Conference on Evolutionary Computa-
tion, pages 665–669, IEEE Press, Piscataway, New Jersey.
Mahfoud, S. W. (1995). Niching Methods for Genetic Algorithms Ph.D. thesis, Department of General
Engineering, IlliGAL Report 95001, University of Illinois at Urbana-Champaign, Urbana,
Illinois.
Maniezzo, V. (1994). Genetic evolution of the topology and weight distribution of neural net-
works. IEEE Transactions on Neural Networks, 5(1):39–53.
Montana, D. J. and Davis, L. (1989). Training feedforward neural networks using genetic algo-
rithms. In Sridharan, S., editor, Proceedings of the Eleventh International Joint Conference on
Artificial Intelligence, pages 762–767, Morgan Kaufmann, San Francisco, California.
Moriarty, D. E. (1997). Symbiotic Evolution of Neural Networks in Sequential Decision Tasks. Ph.D. the-
sis, Department of Computer Sciences, The University of Texas at Austin. Technical Report
UT-AI97-257.
Moriarty, D. E. and Miikkulainen, R. (1997). Forming neural networks through efficient and adap-
tive co-evolution Evolutionary Computation, 5(4):373–399.
Potter, M. A. and De Jong, K. A. (1995). Evolving neural networks with collaborative species.
In Oren, T. I. and Birta, L. G., editors, Proceedings of the 1995 Summer Computer Simulation
Conference, pages 340–345, Society for Computer Simulation, San Diego, California..
Potter, M. A., De Jong, K. A., and Grefenstette, J. J. (1995). A coevolutionary approach to learning
sequential decision rules. In Eshelman, L. J., editor, Proceedings of the Sixth International Con-
ference on Genetic Algorithms, pages 366–372, Morgan Kaufmann, San Francisco, California.
Pujol, J. C. F. and Poli, R. (1998). Evolving the topology and the weights of neural networks using
a dual representation. Special Issue on Evolutionary Learning of the Applied Intelligence Journal,
8(1):73–84.
Radcliffe, N. J. (1993). Genetic set recombination and its application to neural network topology
optimisation. Neural Computing and Applications, 1(1):67–90.
Radding, C. M. (1982). Homologous pairing and strand exchange in genetic recombination. An-
nual Review of Genetics, 16:405–437.
Rosin, C. D. and Belew, R. K. (1997). New methods for competitive evolution. Evolutionary Com-
putation, 5(1):1–29.
Saravanan, N. and Fogel, D. B. (1995). Evolving neural control systems. IEEE Expert, 10(3):23–27.
Schaffer, J. D., Whitley, D., and Eshelman, L. J. (1992). Combinations of genetic algorithms and
neural networks: A survey of the state of the art. In Whitley, D. and Schaffer, J., editors,
Proceedings of the International Workshop on Combinations of Genetic Algorithms and Neural Net-
works (COGANN-92), pages 1–37, IEEE Press, Piscataway, New Jersey.
Sigal, N. and Alberts, B. (1972). Genetic recombination: The nature of a crossed strand-exchange
between two homologous DNA molecules. Journal of Molecular Biology, 71(3):789–793.
Spears, W. (1995). Speciation using tag bits. In Handbook of Evolutionary Computation. IOP Publish-
ing Ltd. and Oxford University Press, Oxford, UK.
Thierens, D. (1996). Non-redundant genetic coding of neural networks. In Proceedings of the IEEE
International Conference on Evolutionary Computation, pages 571–575, IEEE Press, Piscataway,
New Jersey.
Watson, J. D. et al. (1987). Molecular Biology of the Gene Fourth Edition. The Benjamin Cummings
Publishing Company, Inc., Menlo Park, California.
Whitley, D. et al. (1993). Genetic reinforcement learning for neurocontrol problems. Machine
Learning, 13:259–284.
Wieland, A. (1991). Evolving neural network controllers for unstable systems. In Proceedings of
the International Joint Conference on Neural Networks, pages 667–673, IEEE Press, Piscataway,
New Jersey.
Yao, X. (1999). Evolving artificial neural networks. Proceedings of the IEEE, 87(9):1423–1447.
Yao, X. and Liu, Y. (1996). Towards designing artificial neural networks by evolution. Applied
Mathematics and Computation, 91(1):83–90.
Yao, X. and Shi, Y. (1995). A preliminary study on designing artificial neural networks using co-
evolution. In Proceedings of the IEEE Singapore International Conference on Intelligent Control
and Instrumentation, pages 149–154, IEEE Singapore Section.
Zhang, B.-T. and Mühlenbein, H. (1993). Evolving optimal neural networks using genetic algo-
rithms with Occam’s razor. Complex Systems, 7:199–220.