Evolving Neural Networks

through Augmenting
Topologies

Daniele Loiacono

Riconoscimento di segnali con reti neuronali e/o logica sfumata

24/04/2008

Daniele Loiacono
 Neuroevolution

ƒ The artificial evolution of neural networks using

an evolutionary algorithm.

ƒ The network can compute arbitrarily complex

functions, can both learn and perform in the
presence of noisy inputs, and generalize its
behavior to previously unseen inputs.

Daniele Loiacono
Outline

ƒ Motivations

ƒ Introduction to Genetic Algorithms

ƒ Neuroevolution

ƒ NEAT

ƒ Applications

Daniele Loiacono
Motivations to neuroevolution

ƒ Weights of NNs can be learned from a set of

samples <x,y>, in order to minimize the network
error
ƒ Many algorithms have been introduced in the
literature (e.g. error backpropagation)
ƒ Unfortunately:
• we do not have always a set of samples
• learning can get stuck in local minima

Daniele Loiacono
Design of network topology

ƒ What is the best network topology for solving a given

problem?
ƒ A key factor for a successful application of NNs!
ƒ Unfortunately is a trial and error process!

Daniele Loiacono
Problem Dimensionality

ƒ Each connection (weight) in the network is a dimension in

a search space

21-dimensional space 3-dimensional space

ƒ The dimension of search space matters!

ƒ Topology defines the space!

Daniele Loiacono
High Dimensional Space is Hard to Search

ƒ A rough idea:
• 3 dimensional – easy
• 100 dimensional – need a good optimization method
• 10,000 dimensional – very hard
• 1,000,000 dimensional – very very hard
• 100,000,000,000,000 dim. – forget it
ƒ Unfortunately, most interesting problems have
solutions in high dimensional space
ƒ How to get there?

Daniele Loiacono
Outline

ƒ Motivations

ƒ Introduction to Evolutionary Computation

ƒ Neuroevolution

ƒ NEAT

ƒ Applications

Daniele Loiacono
Evolutionary Computation

ƒ Several approaches:

• Genetic Algorithms (Holland 1960s)

• Evolutionary Strategies (Rechenberg 1965)
• Evolutionary Programming (Fogel 1966)
• Genetic Programming (Smith 1980,Koza 1982)

ƒ What are the basic ideas behind them?

Daniele Loiacono
The population

David Brogan, 2006

Daniele Loiacono
Ranking by fitness

David Brogan, 2006

Daniele Loiacono
Mate Selection

Selection
probability is
usually
proportional
to fitness

David Brogan, 2006

Daniele Loiacono
Crossover

Exploit goodness
of
parents

David Brogan, 2006

Daniele Loiacono
Mutation

Explore
unknown

David Brogan, 2006

Daniele Loiacono
Best solution

David Brogan, 2006

Daniele Loiacono
 David Brogan, 2006
Daniele Loiacono
 Important Concepts

ƒ Introduction to Evolutionary Computation

• Genotype and Phenotype
• Fitness
• Selection
• Mutation and Recombination
• Population Diversity
• Building Blocks

Daniele Loiacono
Genotype and phenotype

ƒ The genotype is a representation of the solution

itself, called phenotype
ƒ The process of creating the phenotype from the
genotype is called the genotype to phenotype
mapping
ƒ Mapping can happen in many ways!

Daniele Loiacono
Genotype and phenotype

Genotype

Phenotype

Daniele Loiacono
Genotype and phenotype

Genotype

0011 0111 1010

Phenotype

f ( x) = 3x + 7 x − 10
2

Daniele Loiacono
Genotype
<x1,x2,x3,x4,x5,x6,x7,x8,x9,x10,x11,x12>

Phenotype

Daniele Loiacono
Fitness

ƒ The phenotype of each solution in the population

is evaluated

ƒ The evaluation is performed through a fitness

function.

ƒ Fitness tells which genotypes are better than

others

Daniele Loiacono
Selection

ƒ The selection process defines which solutions

should reproduce (who should be a parent and
who should not)
ƒ Selection is usually based on fitness:
• Roulette Wheel
• Tournament Selection
• Truncation
• Elitism (reproduction of champs)

Daniele Loiacono
Mutation and Recombination

ƒ Mutation
• changes the genotype randomly
• source of diversity and discovery
• a kinf of “local search”

ƒ Recombination
• combines different genotypes
• should be aware of structures of solution !

Daniele Loiacono
Diversity

ƒ When a single genotype dominates the

population, it is converged
ƒ Convergence is premature if a suitable solution
has not yet been found
ƒ How to maintain the diversity?
ƒ Speciation: population is divided into species
• avoid mating and competition inter-species
• protect innovative solution
ƒ Many implementations
• Islands
• Fitness sharing
• Crowding

Daniele Loiacono
Speciation

ƒ A population can be divided into species

ƒ Can prevent incompatibles from mating
ƒ Can protect innovative concepts in niches
ƒ Maintains diversity
ƒ Many methods
• Islands
• Fitness sharing
• Crowding

Daniele Loiacono
Building Blocks

x x

Competent GAs
Daniele Loiacono
Building Blocks

Building Blocks Hypothesis (Goldberg 1989)

ƒ (Competent) GAs can effectively recombine

Building Blocks to find optimal solution
ƒ Recombination plays a key role in Building Blocks
recombination
ƒ (Competent) GAs exploits an implicit parallelism
working on different Building Blocks at the same
time

Daniele Loiacono
Outline

ƒ Motivations

ƒ Introduction to Genetic Algorithms

ƒ Neuroevolution

ƒ NEAT

ƒ Experimental results

Daniele Loiacono
Neuroevolution (NE)

ƒ Artificial evolution of NN using GA

ƒ Fitness is the evaluation of NN performance
ƒ Advantages:
• no need for set of samples
• avoid local minima
ƒ Evolving what?
• weights in a fixed network topology
• topology along with weights (TWEANN)

Daniele Loiacono
TWEANN issues

ƒ Encoding
• direct
• indirect
ƒ Mating (Crossover)
• competing conventions
• free topology and the Holy Grail
ƒ Protecting Innovations
ƒ Initialization and topology minimization
ƒ Examples of TWEANN systems

Daniele Loiacono
TWEANN Encoding

ƒ How to encode a neural network?

ƒ Direct encoding
• genome specifies explicitly the phenotype
• many encoding proposed (from binary
encoding to graph encoding)
ƒ Indirect Encoding
• genome specify how to build the network
• allow for a more compact representation
• can be biased

Daniele Loiacono
Mating in TWEANN

ƒ How can we mate two networks in meaningful

way?
ƒ Challenging even for small networks with the
same topology:

n1 ļ n’1
n1 n2 n3 n’1 n’2 n’3
; n2 ļ n’2
n3 ļ n’3

Daniele Loiacono
Mating in TWEANN

ƒ How can we mate two networks in meaningful

way?
ƒ Challenging even for small networks with the
same topology:

A B CA C B C A
…
Competing Conventions Problem

Daniele Loiacono
Mating in TWEANN

ƒ For fixed and constrained topologies the

competing conventions problem can be solved
with tailored encoding
ƒ But with free topology?

The Holy Grail (Radcliffe, 1993)

Daniele Loiacono
Protecting innovations

ƒ Innovations usually involves larger networks with

more connections
ƒ Larger networks require more time to be
optimized and cannot compete with smaller ones
(at least in the short run)
ƒ Speciation is an effective technique to avoid
competition and mating between networks too
different

Daniele Loiacono
Initialization and topology minimization

ƒ Initial population of random networks

• networks not connected
• heterogeneous networks to mate
ƒ How can network topology be minimized?
• networks bigger than necessary are expensive to
optimize
• fitness penalty

Daniele Loiacono
Structured Genetic Algorithm (sGA) Dasgupta and
McGregor (1992)

ƒ Binary encoding: a bit string represents the

connective matrix of networks
ƒ Advantages
• Evolution can be performed with a standard GA
ƒ Drawbacks
• Computational space is O(n^2) with n number of nodes
• Number of nodes limited

Daniele Loiacono
 Breeder Genetic Programming (Zhang and Muhlenbein)

ƒ Encoding with tree

ƒ Only crossover adapts topology
ƒ Fitness has an explicit penalty term for bigger
networks
ƒ How do we set the penalty? Does it depend from
the problem?

Daniele Loiacono
 Parallel Distributed Genetic Programming (PDGP)
Pujol and Poli (1997)

ƒ Dual encoding:
• linear genome for mutating weights
• Graph representation for a subgraph swapping
crossover
ƒ Does swapping crossover guarantee a good
mating?

Daniele Loiacono
GeNeralized Acquisition of Recurrent Links (GNARL)
Angeline, Saunders, and Pollack (1993)

ƒ Graph Encoding
ƒ Avoid mating because considered disruptive

Daniele Loiacono
Cellular Encoding
Gruau (1993, 1996)

ƒ Indirect encoding (developmental)

ƒ Biased search in topology space
ƒ Mating results are unpredictable
ƒ More compact representation
ƒ Effective but not always more effective than direct
encoding

Daniele Loiacono
Enforced SubPopulations (ESP)
Gomez and Miikkulainen (1997,1999)

ƒ Fixed topology
ƒ When it fails starts from scratch with a new
random topology
ƒ Faster than Cellular Encoding!

Daniele Loiacono
Outline

ƒ Motivations

ƒ Introduction to Genetic Algorithms

ƒ Neuroevolution

ƒ NEAT

ƒ Applications

Daniele Loiacono
NEAT

ƒ NeuroEvolution of Augmenting Topology solve

effectively the TWEANN issues thanks to:
• Historical Markings
• Speciation
• Complexification

Daniele Loiacono
Genetic Encoding in NEAT

Daniele Loiacono
Topological Innovation

Daniele Loiacono
Link Weights Mutation

ƒ A random number is added or subtracted from the

current weight/parameter
ƒ The number can be chosen from uniform,
Gaussian (normal) or other distributions

Daniele Loiacono
 Topology Matching Problem

ƒ When mating networks they may have gene with

the same functionality but in different positions
ƒ In nature homologous genes are aligned and
matched (synapsis)
ƒ How can we solve this problem ?

Daniele Loiacono
 Historical Markings

• Two genes with the same history are

expected to be homologous
Daniele Loiacono
Artificial Synapsis in NEAT

Daniele Loiacono
Speciation in NEAT

ƒ At the beginning topological innovation may not be an

advantage
ƒ In order to let survive useful innovations, they have to be
protected
ƒ NEAT solves this problem with speciation:
• similar networks are clustered
• competition and mating is restricted to networks with the same
space
• fitness sharing prevent from a single species taking over the whole
population

Daniele Loiacono
 Network similarity in NEAT

• Based on historical markings

• A compatibility distance is computed on
the basis of the excess (E), disjoint (D) and
the average weight distance W of matching
genes:
c1E c2 D
δ= + + c3W
N N

Daniele Loiacono
Fitness Sharing in NEAT

ƒ The fitness of each network is normalized with respect to the number

of networks of the same species
ƒ At each species is assigned a number of offspring in the next
generation proportional to the fitness sum of the species members.
ƒ Offspring are generated for each species by mating the fittest r%
members of the species
ƒ Speciation gives to promising topology the chance of being optimized
without the need of competing with all the other networks
ƒ Fitness sharing prevents from a single species taking over the whole
population (i.e. it keeps diversity)

Daniele Loiacono
Complexification

ƒ In NEAT population is initialized with networks

having the identical minimal topology (all inputs
directly connected to outputs)
ƒ More complex topology are introduced with
mutation and survives only if useful
ƒ Complexification has two advantages:
• make trivial the initialization of the population
• keeps limited the dimensionality of search space
through the whole evolutionary process

Daniele Loiacono
Outline

ƒ Motivations

ƒ Introduction to Genetic Algorithms

ƒ Neuroevolution

ƒ NEAT

ƒ Applications

Daniele Loiacono
 NEAT Applications

ƒ NEAT makes new applications possible and has

set performance records

Double Pole Balancing Record

Vehicle Warnings
Keepaway Record

NERO: Real-time Neuroevolution Robot Duel

in a Video Game
Go

Daniele Loiacono
Conclusions

ƒ NEAT is an effective approach to NE thanks to

• Historical marking
• Speciation
• Complexification
ƒ Directions
• Developmental Encoding (HyperNEAT, 2008)
• Hybrid Approach
• Ensembles

Daniele Loiacono
 XCSF with Neural Prediction

6WDWH$FWLRQ *$RQ11
6SDFH 11

11
11
11
11
11

11

11

*$

Daniele Loiacono
 XCSF with Neural Prediction

6WDWH$FWLRQ *$RQ11
6SDFH 11

11
11
11
11
11

11

11

*$

Daniele Loiacono
 XCSF with Neural Prediction

6WDWH$FWLRQ *$RQ11
6SDFH 11

11
11
11
11
11

11

*$

Daniele Loiacono
References

ƒ X. Yao, (1999), “Evolving artificial neural

networks”
ƒ Kenneth O. Stanley and Risto Miikkulainen
(2002). "Evolving Neural Networks Through
Augmenting Topologies". Evolutionary
Computation
ƒ Kenneth O. Stanley and Risto Miikkulainen
(2002). "Efficient Reinforcement Learning
Through Evolving Neural Network Topologies".
Proceedings of the Genetic and Evolutionary
Computation Conference

Daniele Loiacono