2011 10th Mexican International Conference on Artificial Intelligence

Comparison of PSO and DE for training neural

networks
Espinal Andrés, Sotelo-Figueroa Marco, Soria-Alcaraz Jorge A, Ornelas Manuel,Puga Hector, Carpio Martı́n,
Baltazar Rosario, Rico J.L.
División de Estudios de Posgrado e Investigación,
Instituto Tecnológico de León
León, Guanajuato, México
espinalandres86@hotmail.com, masotelof@gmail.com, soajorgea@gmail.com

Abstract—The use of computational resources required for
Feed-Forward Artificial Neural Network (FFANN) training phase
by means of classical techniques such as the backpropagation
learning rule can be prohibitive in some applications. A good
training phase is needed for a high performance of a neural
network. In searching for alternative methods for training phase
of FFANN, some metaheuristic techniques have been used to
do this task. This paper compares the performance of Particle
Swarm Optimization (PSO) and Differential Evolution (DE) as
training methods for FFANN under several well-known pattern
recognition instances.
Keywords-Neural Networks, Particle Swarm Optimization, Dif-
ferential Evolution
A. Feed-Foward Artificial Neural Network
In FFANN, the first term feedforward describes how this
neural network processes and recalls patterns. In a feedforward
neural network, neurons are only connected forward. Each
layer of the neural network contains connections to the next
layer, but there are no back connections [2].
The feed-forward process can be explained as follows. Let
be expression 1 the notation for a feedforward neural network
[3]
W 1 ,b1 W 2 ,b2 W L ,bL
x0 → x1 → · · · → xL (1)

I. I NTRODUCTION where xl ∈ Rnl for all l = 0, . . . , L and W l is an nl × nl−1

Feed-Foward Artificial Neural Network (FFANN) [1] [2]
is a common type of neural network used in a wide range
of classification problems. Commonly this kind of neural
network needs to be trained in order to perform a good
classification rate. This training phase usually means the use
of the backpropagation algorithm. However this algorithm
has shown some disadvantages related with the computational
resources used in some applications[3] . Currently some new
approaches have emerged based in this idea. These approaches
commonly describe the adaptation of metaheuristic techniques
for the FFANN training phase[4][5]. This paper compares the
performance of Particle Swarm Optimization (PSO) [6] and
Diferential Evolution (DE) [7] applied as training methods
for FFANN.
PSO and DE were implemented for the training phase of
FFANN. Our benchmark tests were training process from the
Ionosphere, Irish Plant, Glass, Teaching Assistant Evaluation
and Wine datasets, which were taken from UCI Machine
Learning Repository. A fixed number of function points were
performed and by the training error we were capable to discern
the performance of the training metaheuristics. The paper is
divided as follows: section 2 shows the theory of our approach.
Section 3 exposes the methodology proposed. In section 4 is
explained the experiments and results. Section 5 discusses the
conclusions.
II. C ONCEPTS AND D EFINITIONS
In this section we present the theory of FFANN, PSO and
DE, which are essential for the development of this work.
matrix for all l = 1, . . . , L. There are L + 1 layers of neurons,
and L layers of synaptic weights.
Foward pass. The input vector x0 is transformed into the
output vector xL , by means of a feedforward process by
evaluating the equation 2.
⎛ ⎞

nl−1
xi = f (ui ) = f ⎝
l l
Wij xj + bi ⎠
l l−1 l
(2)
j=1
for l = 1 to L. Where f (uli ) is the evaluation for the input uli
of the xli neuron by an activation function.
B. Particle Swarm Optimization
The Particle Swarm Optimization (PSO) [6] [8] [9]
is a metaheuristic inspired in flocks of birds or schools
of fish. It was developed by J. Kennedy and R. Eberhart
in 1995. It is based on a concept called social metaphore.
This metaheuristics simulates a society where all individuals
contribute with their knowledge to obtain a better solution.
In this metaheuristic each individual is called particle and
moves through a multidimensional space that represents the
social space or search space. The dimension of space depends
on the variables used to represent the problem.
In the search space, the position of each particle is updated
by using its current location and its velocity vector; this vector
tells how fast the particle will move. The eq. 3 calculates the
velocity vector, the eq. 4 is the way to estimate a constriction
coefficient and the eq. 5 updates the position of a particle.

978-0-7695-4605-6/11 $26.00 © 2011 IEEE 83

DOI 10.1109/MICAI.2011.16
This file was provided to the participants of MICAI 2011 courtesy of IEEE CS CPS.
Do not distribute. Other users should download the paper from ieeexplore.ieee.org.
 vi = χ(vi + ϕ1 ∗ (xi − BGlobal ) + ϕ2 ∗ (xi − BLocal )) (3)
2 This vector can be considered as the chromosomes or
χ=  (4)
ϕ−2+ ϕ2 − 4ϕ
xi = xi + v i (5)
where:
• vi is the velocity of the i-th particle.
• xi is the position of the i-th particle.
• BGlobal is the best position found so far by all particles.
• BLocal is the best position found by the i-th particle.
• ϕ1 determines the magnitude of the forces in the direction
of neighbourhood best BGlobal .
• ϕ2 determines the magnitude of the forces in the direction
of personal best BLocal .
• ϕ is a parameter for the constriction coefficient where
ϕ = ϕ1 + ϕ2 > 4.
The algorithm 1 shows the PSO metaheuristic.
Algorithm 1 Particle Swarm Optimization Algorithm
Require: ϕ1 neighbourhood memory coefficient, ϕ2 personal
memory coefficient, n swarm size.
1: Start the swarm particles.
2: Start the vi for each particle in the swarm.
3: while stopping criterion not met do
4: for i = 1 to n do
5: If the i-particle’s fitness is better than that of the
BLocal then replace the BLocal with the i-particle.
6: If the i-particle’s fitness is better than that of the
BGlobal then replace the BGlobal with the i-particle.
7: Update the vi by eq. 3.
8: Update the xi by eq. 5.
9: end for
10: end while
11: return BGlobal
C. Differential Evolution
Differential Evolution (DE) [7] was developed by R.
Storn and K. Price in 1996. It is a vector-based evolutionary
algorithm, and it can be considered as a further development
to genetic algorithm. It is a stochastic search algorithm with
self-organizing tendency and does not use the information of
derivatives [10].
Like in genetic algorithms [11], the design parameters in
a d-dimensional search space are represented as vectors, and
various genetic operators are operated over their bit of strings.
However, unlikely genetic algorithms, differential evolution
carries out operations over each component (each dimension
of solution.)
For a d-dimensional optimization problem with d param-
eters, a population of n solutions are initially generated, so
we have xi solution vectors where i = 1, 2, . . . , n. For each
solution xi at any generation t, we use the conventional
notation as
xti = (xt1,i , xt2,i , . . . , xtd,i ) (6)
84
which consist of d-components in the d-dimensional space.
genomes.
This metaheuristic consists of two main steps: mutations
and selection.
Differential evolution has several schemes for carrying out
the mutation step which generates a so-called donor vector.
One of these schemes is know as DE/Current to best/1 which
is showed in eq. 7.
vit+1 = F (xtbest − xti ) + F (xtp − xtq ) (7)
where vit+1 is the donor vector, F ∈ [0, 2] is a parameter
often referred to as the differential weight, xtbest is the genome
with the best fitness value in the population, xti is the current
genome and, xtp and xtq are two genomes randomly chosen
(indexes i, p and q must be different each other).
Selection is essentially the same as that used in genetic
algorithms. It consists on to select the best fitness. For a
minimization problem, the minimum objective value must be
selected. Therefore, we have
 t+1
vi if f (vit+1 ) ≤ f (xti )
xt+1 = (8)
i xti otherwise
All the above two components can be seen in the pseudo
code shown in algorithm 2. The overall search efficiency is
controlled by the differential weight F .
The algorithm 2 shows the DE metaheuristic.
Algorithm 2 Differential Evolution Algorithm
Require: n population size.
1: Initializate the initial population.
2: Set the weight F ∈ [0, 2]
3: while stopping criterion not met do
4: for i = 1 to n do
5: For each xti randomly choose 2 distinct genomes xtp
and xtq , the vector xtbest is the genome with the best
fitness value in the population.
6: Generate a new donor vector vit+1 by eq. 7 and
update the genome xti by eq. 8.
7: end for
8: end while
9: return xtbest , the best genome in the last run, like the
solution vector.
III. M ETHODOLOGY PROPOSED
The methodology evaluates two FFANN by means of PSO
and DE in the training phase. For each test intance, we use
the same architecture for both FFANN.
Basically the idea to achieve the FFANN training process
by using PSO and DE metaheuristics consists on to work in
a multi - dimensional search space and try to minimize a
learning measure of the FFANN. The dimension of the search
space needs to be equal to the amount of neural weights. The
figure 1 shows the transformation of the FFANN to a particle
or genome.
 The training phase was performed in the batch training
approach [4]. It means that for all patterns of the training
dataset, the quadratic error [3] is calculated from the target
output and the neural network output (eq. 9) before to adjust
all weights of the neural network.

M 
m
(p) (p)
Quadratic Error = (tk − yk )2
p=1 k=1
where: M = number of patterns, m = number of
neural outputs, t = target output, y = neural network
output.
Since we have a continuous space, the Quadratic Error can
be considered like a F itness f unction. This function can be
easily implemented in a metaheuristic technique such as PSO
and DE showed in the previous section. The dimension of the
problem will be the weights of the links between nodes in our
FFANN’s model.
The final goal for these two training techniques is the
same that the backpropagation learning rule: To minimize the
quadratic error of the neural network outputs.
Fig. 1. Codification of a Particle/Genome for representing a FFANN
configuration.
IV. E XPERIMENTS AND R ESULTS
In this section we discuss the experiment design, execution
and results of the comparison between PSO and DE for the
training phase.
A. Dataset description
We chose five well-know dataset instances from the UCI
Machine Learning Repository:
Glass: The Glass [12] dataset is an study of classifi-
cation of types of glasses. It was motivated by crim-
inological investigation. This dataset is formed by 7
classes and it has 214 instances with 10 attributes
each one.
Ionosphere: The Ionosphere [13] dataset was col-
lected by a system in Goose Bay, Labrador. This sys-
tem consists of a phased array of 16 high-frequency
antennas with a total transmitted power on the order
of 6.4 kilowatts. The instances in this databse are de-
scribed by 2 attributes per pulse number, correspond-
ing to the complex values returned by the function
resulting from the complex electromagnetic signal.
The dataset has 351 instances with 35 attributes each
one. The first 34 attributes are continuous and the
(9)
35th is the type of class, good or bad. Good radar
returns are those showing evidence of some type of
structure in the ionosphere and bad returns are those
that do not; their signals pass through the ionosphere.
Irish Plant: The Irish Plant [14] dataset contains 3
classes of 50 instances each, where each class refers
to a type of iris plant. One class is linearly separable
from the other 2; the latter are not linearly separable
from each other. The dataset has 150 instances ,50 in
each of three classes. Each instances has 4 attributes:
sepal length in cm, sepal width in cm, petal length
in cm and petal width in cm. The classes are: Iris
Setosa, Iris Versicolour and Iris Virginica.
Teaching Assistant Evaluation (TAE): The Teach-
ing Assistant Evaluation [15] dataset consist of eval-
uations of teaching performance over three regular
semesters and two summer semesters of 151 teaching
assistants assignment at the Statistics Department of
the University of Wisconsin-Madison. The scores
were divided into 3 roughly equal-sized categories
to form the class variable. Each instance is formed
by 5 attributes.
Wine: The Wine [16] dataset is the result of a
chemical analysis of wines grown in the same region
in Italy but derived from three different cultivations.
The analysis determined the quantities of 13 con-
stituents found in each of the three types of wines.
The dataset has 178 instances with 13 attributes each
one.
B. Experimental design
The configuration of the experiments realized for this paper
is described on the next paragraphs.
1) FFANN Configuration: As each dataset has a very par-
ticular structure (number of classes and number of features),
the neural network architecture changed for each dataset. Since
the main goal of this paper is to know which metaheuristic is
the best in the minimization of the quadratic error, the authors
decided to use FFANNs with only one hidden layer for reasons
of simplicity.The number of neurons in the input layer was
given by the number of features of the dataset, the hidden layer
was estimated as the double of the neurons in the input layer
minus one and finally the number of neurons of the output
layer is equal to the number of classes of the dataset. All
hidden and output neurons have the sigmoid function as their
activation function. The eq. 10 shows the sigmoid function.
1
f (uli ) = l (10)
1 + e−ui
where uli is the input of the neuron xli , as it’s explained in
section II-A.
85
 2) Metaheuristics Configuration: In order to gather in-
formation about the performance of the two metaheuritics
proposed (PSO and DE) in the training phase of the FFANN,
we used the quadratic error from the neural network output
and the desired output. Our main objective is to minimize the
difference between the current solution and the desire solution.
Equation 9 was used as the objetive function in the
proposed metaheuristic techniques PSO and DE. This ap-
proach transforms the original problem of classification into
an optimization problem, where we need to find the correct
weights that minimizes the fitness function.
For each dataset we applied 10,000 iterations of the pro-
posed metaheuristics. Finally we compared the last fitness
reported for each heuristic to analyze its performance.
PSO setup: PSO used 20 particles (each particle rep-
Fig. 2. Training/Optimization Process for Glass dataset
resents a complete FFANN array of weights), with a
χ parameter of autoadaptation. The parameter of the
neighbourhood memory coefficient is ϕ1 = 2.05, and
for the personal memory coefficient is ϕ2 = 2.05.
DE setup: DE used 20 individuals (same as PSO,
each individuals represents the complete set of
weights of a FFANN), and F = 0.15. The scheme
for the mutation step used in this work was the
DE/Current to best/1 scheme (eq. 7).
We applied this design for each dataset in order to achieve
a fair comparison between PSO and DE.

C. Results
Once the test was executed for each dataset with the
proposed metaheuristics, we achieved the results shown in
table I. Figures 2 to 6 show the training/optimization process
for each dataset done by both metaheuristics: PSO and DE.

Dataset PSO DE
Fig. 3. Training/Optimization Process for Ionosphere dataset
Glass 57.881 133.51
Iono 0.362 12.574
Iris 1.49e−3 2.00
TAE 48.67 78.30
Wine 0.003 58.054
TABLE I
F INAL Q UADRATIC ERROR FROM EACH DATASET

These results can be analyzed by simple visual inspection.

It’s obvious that PSO gets a better performance than DE as a
training method for FFANN.

D. Discussion
The PSO algorithm is clearly the winner against the DE
algorithm, but we need to consider the number of operations
of both algorithms.
The number of operations of each algorithms affects directly
the computational resources used i.e a higher number of oper- Fig. 4. Training/Optimization Process for Iris dataset
ations produces a major resources consumption. This situation
is relevant for the results. The DE algorithm is simpler that
the PSO algorithm by the next reasons:

86

Fig. 5. Training/Optimization Process for TAE dataset
Fig. 6. Training/Optimization Process for Wine dataset
• DE makes a mutation by means of a vectorial subtraction,
and then evaluates the fitness function in order to decide
if the new solution replaces the old one.
• PSO needs to update each particle velocity.
• PSO requires more complex operations to generate a new
solution i.e needs to evaluate local and global knowledge
to discern a new solution.
• PSO checks for each particle if another particle has found
a better solution that the current reported as BGlobal and
GLocal .
If we consider each difference, we can assume that the PSO
algorithm at least uses a twice of computational resources that
the DE algorithm. This excess in the operations can be the
reason of its better performance.
V. C ONCLUSIONS
This paper has compared two metaheuristcs for FFANN’s
training phase. We have used PSO and DE as training phase.
DE have shown a simpler computing performance since the
number of operation made for this algorithm is at least half
the number of operations made for PSO. The test shows that
PSO works in a better way than DE in the minimization of
quadratic error function as fitness function.
87
This paper only works on the training phase, so for further
work, we propose to complete the process of classification in
order to validate that a minimum error in the training phase
equals to a better performance of classification rate.
ACKNOWLEDGEMENT
Authors thank the support received from the CONACYT
and DGEST (Grant 3528.10-P).
R EFERENCES
[1] E. P. P. A. Derks, M. S. S. Pastor, and L. M. C. Buydens, “Robustness
analysis of radial base function and multi-layered feed-forward
neural network models,” Chemometrics and Intelligent Laboratory
Systems, vol. 28, no. 1, pp. 49 – 60, 1995. [Online]. Available:
http://www.sciencedirect.com/science/article/pii/016974399580039C
[2] J. Heaton, Introduction to Neural Networks for Java, Second Edition.
Heaton Research, Inc., 2008.
[3] M. Friedman and A. Kandel, Introduction to pattern recognition statis-
tical, structural, neural and fuzzy logic approaches. World Scientific,
2000.
[4] V. G. Gudise, G. K. Venayagamoorthy, and S.-M. /eee, “Comparison of
particle swarm optimization and backpropagation as training algorithms
for neural networks,” in in Proceedings of the IEEE Swarm Intelligence
Symposium 2003 (SIS 2003, 2003, pp. 110–117.
[5] J. Ilonen, J.-K. Kamarainen, and J. Lampinen, “Differential evolution
training algorithm for feed-forward neural networks,” Neural Processing
Letters, vol. 17, no. 1, pp. 93–105, 2003.
[6] J. Kennedy and R. C. Eberhart, “Particle swarm optimization,” IEEE
Int. Conf. Neural Netw, vol. 4, pp. 1942–1948, 1995.
[7] R. Storn and K. Price, “Differential evolution - a simple and
efficient heuristic for global optimization over continuous spaces,” J. of
Global Optimization, vol. 11, pp. 341–359, December 1997. [Online].
Available: http://portal.acm.org/citation.cfm?id=596061.596146
[8] C. Maurice, Particle Swarm Optimization. USA: Wiley-ISTE, 2006.
[9] R. Poli, J. Kennedy, and T. Blackwell, “Particle swarm optimization,”
Swarm Intelligence, vol. 1, no. 1, pp. 33–57, Jun. 2007. [Online].
Available: http://dx.doi.org/10.1007/s11721-007-0002-0
[10] X. S. Yang, Nature Inspired Metaheuristic Algorithms, 2nd ed. Luniver
Press, 2008.
[11] J. Holland, “Adaptation in natural and artificial systems,” University of
Michigan Press, 1975. [Online]. Available: http://mitpress.mit.edu/
[12] P. Zhong and M. Fukushima, “A regularized nonsmooth newton method
for multi-class support vector machines,” in Systems Analysis, Optimiza-
tion and Data Mining in Biomedicine. Taylor and Francis, Mar 2007,
vol. 22, pp. 225–236.
[13] V. G. Sigillito, S. P. Wing, L. V. Hutton, and K. B. Baker, “Classification
of radar returns from the ionosphere using neural networks,” Johns
Hopkins APL Technical Digest, vol. 10, pp. 262–266, 1989.
[14] R. A. FISHER, “The use of multiple measurements in taxonomic
problems,” Annals of Human Genetics, vol. 7, no. 2, pp.
179–188, 1936. [Online]. Available: http://dx.doi.org/10.1111/j.1469-
1809.1936.tb02137.x
[15] W. Loh and Y. Shih, “Split selection methods for classification trees,”
Statistica Sinica, 1997.
[16] K. Ali and M. Pazzani, “Error reduction through learning multiple
descriptions, in press,” Machine Learning, vol. 24, 1996.