1812 D Data-Mining and Knowledge Discovery, Neural Networks in
that is, a mapping from observations about an item to con- chine learning method which has to be considered when-
clusions about its target value. Their article concentrates
on the machine learning view.
Orlov, Sipper and Hauptman (see  Genetic and
Evolutionary Algorithms and Programming: General In-
troduction and Application to Game Playing) cover ge-
netic and evolutionary algorithms, which are a family of
search algorithms inspired by the process of (Darwinian)
evolution in Nature. Common to all the different fam-
ily members is the notion of solving problems by evolv-
ing an initially random population of candidate solu-
tions, through the application of operators like crossover
and mutation inspired by natural genetics and natural se-
lection, such that in time “fitter” (i. e., better) solutions
emerge. The field, whose origins can be traced back to
the 1950s and 1960s, has come into its own over the past
two decades, proving successful in solving multitudinous
problems from highly diverse domains including (to men-
tion but a few): optimization, automatic programming,
electronic-circuit design, telecommunications, networks,
finance, economics, image analysis, signal processing, mu-
sic, and art.
Berkhin and Dhillon (see  Knowledge Discovery:
Clustering) discuss the condition where data found in sci-
entific and business applications usually do not fit a partic-
ular parametrized probability distribution. In other words,
the data are complex. Knowledge discovery starts with
exploration of this complexity in order to find inconsis-
tencies, artifacts, errors, etc. in the data. After data are
cleaned, it is usually still extremely complex. Descriptive
data mining deals with comprehending and reducing this
complexity. Clustering is a premier methodology in de-
scriptive unsupervised data mining. A cluster could rep-
resent an important subset of the data such as a galaxy in
astronomical data or a segment of customers in market-
ing applications. Clustering is important as a fundamental
technology to reduce data complexity and to find data pat-
terns in an unsupervised fashion. It is universally used as
a first technology of choice in data exploration.
Džeroski, Panov and Zenko (see  Machine Learning,
Ensemble Methods in) cover ensemble methods, which are
machine learning methods that construct a set of predic-
tive models and combine their outputs into a single predic-
tion. The purpose of combining several models together is
to achieve better predictive performance, and it has been
shown in a number of cases that ensembles can be more
accurate than single models. While some work on ensem-
ble methods has already been done in the 1970s, it was not
until the 1990s, and the introduction of methods such as
bagging and boosting that ensemble methods started to be
more widely used. Today, they represent a standard ma-
ever good predictive accuracy is demanded.
Liu and Zhao (see  Manipulating Data and Dimen-
sion Reduction Methods: Feature Selection) cover feature
selection, which is the study of algorithms for reducing
dimensionality of data for various purposes. One of the
most common purposes is to improve machine learning
performance. The other purposes include simplifying data
description, streamlining data collection, improving com-
prehensibility of the learned models, and helping gain in-
sight through learning. The objective of feature selection is
to remove irrelevant and/or redundant features and retain
only relevant features. Irrelevant features can be removed
without affecting learning performance. Redundant fea-
tures are a type of irrelevant features. The distinction is
that a redundant feature implies the co-presence of an-
other feature; individually, each feature is relevant, but the
removal of either one will not affect learning performance.
As a plethora of data are generated in every possible means
with the exponential decreasing costs of data storage and
computer processing power, data dimensionality increases
on a scale beyond imagination in cases ranging from trans-
actional data to high-throughput data. In many fields such
as medicine, health care, Web search, and bioinformatics,
it is imperative to reduce high dimensionality such that ef-
ficient data processing and meaningful data analysis can
be conducted in order to mine nuggets from high-dimen-
sional, massive data.
Data-Mining and Knowledge
Discovery, Neural Networks in
MARKUS BRAMEIER
Bioinformatics Research Center, University of Aarhus,
Århus, Denmark
Article Outline
Glossary
Definition of the Subject
Introduction
Neural Network Learning
Feedforward Neural Networks
Backpropagation
Other Learning Rules
Other Neural Network Architectures
Self-organizing Maps
Future Directions
Bibliography
 Data-Mining and Knowledge Discovery, Neural Networks in
Glossary
Artificial neural network An artificial neural network is
a system composed of many simple, but highly inter-
connected processing nodes or neurons which oper-
ate in parallel and collectively. It resembles biological
nervous systems in two basic functions: (1) Experien-
tial knowledge is acquired through a learning process
and can be retrieved again later. (2) The knowledge is
stored in the strength (weights) of the connections be-
tween the neurons.
Artificial neuron An artificial neuron receives a number
of inputs, which may be either external inputs to the
neural network or outputs of other neurons. Each in-
put connection is assigned a weight, similar to the
synaptic efficacy of a biological neuron. The weighted
sum of inputs is compared against an activation level
(threshold) to determine the activation value of the
neuron.
Activation function The activation or transfer function
transforms the weighted inputs of a neuron into
an output signal. Activation functions often have
a “squashing” effect. Common activation functions
used in neural networks are: threshold, linear, sigmoid,
hyperbolic, and Gaussian.
Learning rule The learning rule describes the way a neu-
ral network is trained, i. e., how its free parameters un-
dergo changes to fit the network to the training data.
Feedforward network Feedforward neural networks are
organized in one or more layers of processing units
(neurons). In a feedforward neural network the signal
is allowed to flow one-way only, i. e., from inputs to
outputs. There are no feedback loops, i. e., the outputs
of a layer do not affect its inputs.
Feedback networks In feedback or recurrent networks
signals may flow in both directions. Feedback net-
works are dynamic such that they have a state that is
changing continuously until it reaches an equilibrium
point.
Definition of the Subject
Neural networks (NNs) or, more precisely, artificial neu-
ral networks (ANNs) are mathematical and computational
models that are inspired by the way biological nervous sys-
tems process information. A neural network model con-
sists of a larger number of highly interconnected, simple
processing nodes or units which operate in parallel and
perform functions collectively, roughly similar to biolog-
ical neural networks. ANNs, like their biological counter-
part, are adaptive systems that learn by example. Learning
works by adapting free model parameters, i. e., the signal
D 1813
strength of the connections and the signal flow, to exter-
nal information that is presented to the network. In other
terms, the information is stored in the weighted connec-
tions. If not stated explicitly, let the term “neural network”
mean “artificial neural network” in the following.
In more technical terms, neural networks are non-lin-
ear statistical data modeling tools. Neural networks are
generally well suited for solving problem tasks that in-
volve classification, pattern recognition and decision mak-
ing. The power, and usefulness of neural networks have
been demonstrated in numerous application areas, like
image processing, signal processing, biometric identifica-
tion – including handwritten character, fingerprint, face,
and speech recognition – robotic control, industrial engi-
neering, and biomedicine. In many of these tasks neural
networks outperform more traditional statistical or artifi-
cial intelligence techniques or may even achieve human-
like performance. The most valuable characteristics of
neural networks are adaptability and tolerance to noisy or
incomplete data. Another important advantage is in solv-
ing problems that do not have an algorithmic solution or
for which an algorithmic solution is too complex or time-
consuming to be found.
The first neural network model was developed by Mc-
Culloch and Pitts in the 1940s [32]. In 1958 Rosenblatt [41]
described the first learning algorithm for a single neu-
ron, the perceptron model. After Rumelhart et al. [46] in-
vented the popular backpropagation learning algorithm
for multi-layer networks in 1986, the field of neural net-
works gained incredible popularity in the 1990s.
Neural networks is regarded as a method of machine
learning, the largest subfield of artificial intelligence (AI).
Conventional AI mainly focuses on the development of
expert systems and the design of intelligent agents. To-
day neural networks also belong to the more recent field of
computational intelligence (CI), which also includes evolu-
tionary algorithms (EAs) and fuzzy logic.
Introduction
Herein I provide a concise introduction to the two most
popular neural network types used in application, back-
propagation neural networks (BPNNs) and self-organiz-
ing maps (SOMs). The former learn high-dimensional
non-linear functions from given input–output associa-
tions for solving classification and approximation (regres-
sion) problems. The latter are primarily used for data clus-
tering and visualization and for revealing relationships be-
tween clusters. These models are discussed in context with
alternative learning algorithms and neural network archi-
tectures.
1814 D Data-Mining and Knowledge Discovery, Neural Networks in
Biological Motivation
Artificial neural networks are inspired by biological ner-
vous systems, which are highly distributed and intercon-
nected networks. The human brain is principally com-
posed of a very large number of relatively simple neu-
rons (approx. 100 billion), each of which is connected to
several thousand other neurons, on average. A neuron is
a specialized cell that consists of the cell body (the soma),
multiple spine-like extensions (the dendrites) and a single
nerve fiber (the axon). The axon connects to the dendrites
of another neuron via a synapse. When a neuron is ac-
tivated, it transmits an electrical impulse (activation po-
tential) along its axon. At the synapse the electric signal
is transformed into a chemical signal such that a certain
number of neurotransmitters cross the synaptic gap to the
post synaptic neuron, where the chemical signal is con-
verted back to an electrical signal to be transported along
the dendrites. The dendrites receive signals from the axons
of other neurons. One very important feature of neurons is
that they react delayed. A neuron combines the strengths
(energies) of all received input signals and sends out its
own signal (“fires”) only if the total signal strength exceeds
a certain critical activation level. A synapse can either be
excitatory or inhibitory. Input signals from an excitatory
synapse increase the activation level of the neuron while
inputs from an inhibitory synapse reduce it. The strength
of the input signals critically depends on modulations at
the synapses. The brain learns basically by adjusting num-
ber and strength of the synaptic connections.
Artificial neural networks copy only a small amount of
the biological complexity by using a much smaller num-
ber of simpler neurons and connections. Nevertheless, ar-
tificial neural networks can perform remarkably complex
tasks by applying a similar principle, i. e., the combina-
tion of simple and local processing units, each calculating
a weighted sum of its inputs and sending out a signal if the
sum exceeds a certain threshold.
History and Overview
The history of artificial neural networks begins with a dis-
crete mathematical model of a biological neural network
developed by pioneers McCulloch and Pitts in 1943 [32].
This model describes neurons as threshold logic units
(TLUs) or binary decision units (BDNs) with multiple bi-
nary inputs and a single binary output. A neuron outputs 1
(is activated) if the sum of its unweighted inputs exceeds
a certain specified threshold, otherwise it outputs 0. Each
neuron can only represent simple logic functions like OR
or AND, but any boolean function can be realized by com-
binations of such neurons.
In 1958 Rosenblatt [41] extended the McCulloch–Pitts
model to the perceptron model. This network was based
on a unit called the perceptron, which produces an output
depending on the weighted linear combination of its in-
puts. The weights are adapted by the perceptron learning
rule. Another single-layer neural network that is based on
the McCulloch–Pitts neuron is the ADALINE (ADAptive
Linear Element) which was invented in 1960 by Widrow
and Hoff [55] and employs a Least-Mean-Squares (LMS)
learning rule.
In 1969 Minsky and Papert [33] provided mathemat-
ical proofs that single-layer neural networks like the per-
ceptron are incapable of representing functions which are
linearly inseparable, including in particular the exclusive-
or (XOR) function. This fundamental limitation led the re-
search on neural networks to stagnate for many years, un-
til it was found that a perceptron with more than one layer
has far greater processing power.
The backpropagation learning method was first de-
scribed by Werbos in 1974 [53,54], and further developed
for multi-layer neural networks by Rumelhart et al. in
1986 [46]. Backpropagation networks are by far the most
well known and most commonly used neural networks to-
day.
Recurrent auto-associative networks were first de-
scribed independently by Anderson [2] and Kohonen [21]
in 1977. Invented in 1982, the Hopfield network [17] is
a recurrent neural network in which all connections are
symmetric. All neurons are both input and output neurons
and update their activation values asynchronously and in-
dependently from each other. For each new input the net-
work converges dynamically to a new stable state. A Hop-
field network may serve as an associative, i. e., content-ad-
dressable, memory.
The Boltzmann machine by Ackley et al. [1] can be
seen as an extension of the Hopfield network. It uses
a stochastic instead of a deterministic update rule that sim-
ulates the physical principle of annealing. The Boltzmann
machine is one of the first neural networks to demonstrate
learning of an internal representation (hidden units).
It was also in 1982 when Kohonen first published
his self-organizing maps [22], a neural networks model
based on unsupervised learning and competitive learning.
SOMs produce a low-dimensional representation (feature
map) of high-dimensional input data while preserving
their most important topological features.
Significant progress was made in the 1990s in the field
of neural networks which attracted a great deal of atten-
tion both in research and in many application domains
(see Sect. “Definition of the Subject”). Faster computers al-
lowed a more efficient solving of more complex problems.
 Data-Mining and Knowledge Discovery, Neural Networks in
Hardware implementations of larger neural networks were
realized on parallel computers or in neural network chips
with multiple units working simultaneously.
Characteristics of Neural Networks
Interesting general properties of neural networks are that
they (1) mimic the way the brain works, (2) are able to
learn by experience, (3) make predictions without having
to know the precise underlying model, (4) have a high fault
tolerance, i. e., can still give the correct output to missing,
noisy or partially correct inputs, and (5) can work with
data they have never seen before, provided that the under-
lying distribution fits the training data.
Computation in neural networks is local and highly
distributed throughout the network such that each node
operates by itself, but tries to minimize the overall network
(output) error in cooperation with other nodes. Working
like a cluster of interconnected processing nodes, neural
networks automatically distribute both the problem and
the workload among the nodes in order to find and con-
verge to a common solution. Actually, parallel processing
was one of the original motivations behind the develop-
ment of artificial neural networks.
Neural networks have the topology of a directed graph.
There are only one-way connections between nodes, just
like in biological nervous systems. A two-way relationship
requires two one-way connections. Many different net-
work architectures are used, often with hundreds or thou-
sands of adjustable parameters.
Neural networks are typically organized in layers
which each consist of a number of interconnected nodes
(see Fig. 2 below). Data patterns are presented to the sys-
tem via the input layer. This non-computing layer is con-
nected to one or more hidden layers where the actual pro-
cessing is done. The hidden layers then link to an output
layer which combines the results of multiple processing
units to produce the final response of the network.
The network acts as a high-dimensional vector func-
tion, taking one vector as input and returning another vec-
tor as output. The modeled functions are general and com-
plex enough to solve a large class of non-linear classifica-
tion and estimation problems. No matter which problem
domain a neural network is operating in, the input data al-
ways have to be encoded into numbers which may be con-
tinuous or discrete.
The high number of free parameters in a neural net-
work and the high degree of collinearity between the neu-
ron outputs let individual parameter settings (weight co-
efficients) become meaningless and make the network for
the most part uninterpretable. High-order interactions be-
D 1815
tween neurons also do not permit simpler substructures in
the model to be identified. Therefore, an extraction of the
acquired knowledge from such black box predictors to un-
derstand the underlying model is almost impossible.
Neural Network Learning
The human brain learns by practice and experience. The
learned knowledge can change if more information is re-
ceived. Another important element of learning is the abil-
ity to infer knowledge, i. e., to make assumptions based
on what we know and to apply what we have learned in
the past to similar problems and situations. One theory of
the physiology of human learning by repetition is that re-
peated sequences of impulses strengthen connections be-
tween neurons and form memory paths. To retrieve the
learned information, nerve impulses follow these paths to
the correct information. If we get out of practice these
paths may diminish over time and we forget what we have
learned.
The information stored in a neural network is con-
tained in its free parameters. In general, the network ar-
chitecture and connections are held constant and only the
connection weights are variable during training. Once the
numbers of (hidden) layers and units have been selected,
the free parameters (weights) are set to fit the model or
function represented by the network to the training data,
following a certain training algorithm or learning rule.
A neural network learns, i. e., acquires knowledge, by ad-
justing the weights of connections between its neurons.
This is also referred to as connectionist learning. Train-
ing occurs iteratively in multiple cycles during which the
training examples are repeatedly presented to the network.
During one epoch all data patterns pass through the net-
work once.
The three major learning paradigms include super-
vised learning, unsupervised learning, and reinforcement
learning.
Supervised learning, in general, means learning by an
external teacher using global information. The problem
the network is supposed to solve is defined through a set of
training examples given. The learning algorithm searches
the solution space F, the class of possible functions, for
a function f  2 F that matches this set of input–output
associations (Ex ; yE) best. In other words, the mapping im-
plied by the sample data has to be inferred.
Training a neural network means to determine a set
of weights which minimizes its prediction error on the
training set. The cost or error function E : F ! R mea-
sures the error between the desired output values yE and
the predicted network outputs f (E x ) over all input vec-
1816 D Data-Mining and Knowledge Discovery, Neural Networks in
tors xE. That means, it calculates how far away the current
state f of the network is from the optimal solution f  with
E( f  )  E( f ) 8 f 2 F.
A neural network cannot perfectly learn a mapping if
the input data does not contain enough information to de-
rive the desired outputs. It may also not converge if there
is not enough data available (see also below).
Unsupervised learning uses no external teacher and
only local information. It is distinguished from supervised
learning by the fact that there is no a priori output. In un-
supervised learning we are given some input data xE, and
the cost function to be minimized can be any function
of xE and the network output f (E x ). Unsupervised learn-
ing incorporates self-organization, i. e., organizes the in-
put data by using only their inherent properties to reveal
their emergent collective properties.
A neural network learns offline if learning phase and
operation (application) phase are separated. A neural net-
work learns online if both happens at the same time. Usu-
ally, supervised learning is performed offline, whereas un-
supervised learning is performed online.
In reinforcement learning, neither inputs xE nor outputs
yE are given explicitly, but are generated by the interactions
of an agent within an environment. The agent performs an
action yE with costs c according to an observation xE made
in the environment. The aim is to discover a policy or plan
for selecting actions that minimizes some measure of the
expected total costs.
Overtraining and Generalization
The overall motivation and most desirable property of
neural networks is their ability to generalize to new un-
known data, i. e., to classify patterns correctly on which
they have not been trained. Minimizing the network er-
ror on the training examples only, does not automatically
minimize the real error of the unknown underlying func-
tion. This important problem is called overfitting or over-
training.
A regular distribution of training examples over the in-
put data space is important. Generalization is reasonable
only as long as the data inputs remain inside the range for
which the network was trained. If the training set only in-
cluded vectors from a certain part of the data space, pre-
dictions on other parts are random and likely wrong.
Overtraining may occur also when the iterative train-
ing algorithm is run for too long and if the network is too
complex for the problem to solve or the available quan-
tity of data. A larger neural network with more weights
models a more complex function and invariably achieves
a lower error, but is prone to overfitting. A network with
less weights, on the other hand, may not be sufficiently
powerful to model the underlying function.
A simple heuristic, called early stopping, helps to en-
sure that the network will generalize well to examples not
in the training set. One solution is to check progress dur-
ing training against an independent data set, the valida-
tion set. As training progresses, the training error naturally
decreases monotonically and, providing training is min-
imizing the true error function, also the validation error
decreases. However, if the validation error stops dropping
or even starts to increase again, this is an indication that
the network is starting to overfit the data. Then the opti-
mization process has become stuck in a local minima and
training should be stopped. The weights that produced
the minimum validation error are then used for the final
model.
In this case of overtraining, the size of the network, i. e.,
the number of hidden units and/or hidden layers, may be
decreased. Neural networks typically involve experiment-
ing with a large number of different configurations, train-
ing each one a number of times while observing the vali-
dation error. A problem with repeated experimentation is
that the validation set is actually part of the training pro-
cess. One may just find a network by chance that happens
to perform well on the validation set. It is therefore nor-
mal practice to reserve a third set of examples for testing
the final model on this test set.
In many cases a sufficient amount of data is not avail-
able, however. Then we have to get around this problem by
resampling techniques, like cross validation. In principle,
multiple experiments are conducted, each using a different
division of the available data into training and validation
set. This should remove any sampling bias. For small data
sets, where splitting the data would leave too few observa-
tions for training, leave-one-out validation may be used to
determine when to stop training or the optimal network
size.
Machine-learning techniques, like neural networks,
require both positive and negative training examples for
solving classification problems. Because they minimize an
overall error, the proportion of positive and negative ex-
amples in the training set is critical. Ideally, the relation
should be close to the (usually unknown) real distribution
in the data space. Otherwise, it may bias the network’s de-
cision to be more often wrong on unknown data.
Feedforward Neural Networks
In feedforward neural networks the information is passed
in only one direction (forward) from the inputs, through
the hidden nodes (if any) to the output nodes. There are
 Data-Mining and Knowledge Discovery, Neural Networks in
no connections backwards to neurons of upper layers i. e.,
there are no feedback loops or cycles in the network.
In feedforward networks with a single-layer of weights,
the inputs are directly connected to the output units (sin-
gle-layer neural networks). Multi-layer feedforward net-
works use additional intermediate layers of hidden units.
Neural networks with two or more processing layers may
have far greater processing power than networks with only
one layer. Single-layer neural networks are only capable of
learning linearly separable patterns and functions.
The Perceptron
The most simple kind of feedforward neural network is
w i j (2)
a perceptron, which consists of a single pseudo input layer
and one or more processing nodes in the output layer. All
w i j D ˛(o i  y i )x i j : (3)
inputs are weighted and fed directly to the output neu-
ron(s) (see Fig. 1). Each node calculates the sum of the
products of weights and inputs. If this value is above some
threshold (typically 0) the neuron takes the activated value
1, otherwise it outputs 0. Neurons with this kind of acti-
vation function are called threshold units. In the literature
the term perceptron often refers to networks consisting of
only one (output) neuron.
More formally, the perceptron is a linear binary clas-
sifier that maps a binary n-dimensional input vector
xE 2 f0; 1gn to a binary output value f (w E  xE) 2 f0; 1g cal-
culated as
 to learn, i. e., perfectly classify, linearly separable patterns
1 if s > T
f (s) D
0 otherwise
P layer neural networks with more than one output unit.
where s D w E  xE D niD1 w i x i is the input sum and w
a vector of real-valued weights. The constant threshold T
does not depend on any input value.
In addition to the network topology, the learning rule
is an important component of neural networks. Percep-
trons can be trained by a simple learning algorithm, called
the delta rule or perceptron learning rule. This realizes
Data-Mining and Knowledge Discovery, Neural Networks in, Fig-
ure 1
Principle structure of single-unit perceptron network
D 1817
a simple stochastic gradient descent where the weights of
the network are adjusted depending on the error between
the predicted outputs of the network and the example out-
puts. The delta rule changes the weight vector such that
the output error is minimized. McClelland and Rumel-
hart [30] proved that a neural network using the delta rule
can learn associations whenever the inputs are linearly in-
dependent.
All neurons of a perceptron share the same structure
and learning algorithm. Each weight w i j , representing the
influence of input x i on neuron j, is updated at time t ac-
cording to the rule:
w i j (t C 1) D w i j (t) C
The network learns by updating the weight vector after
each iteration (training example) by an amount propor-
tional to the difference between given output oi and calcu-
lated output y i D f (s i ). The learning rate ˛ is a constant
with 0 < ˛ < 1 and regulates the learning speed.
The training data set is linearly separable in n-dimen-
sional data space if its two-classes of vectors xE can be sep-
arated by an (n  1)-dimensional hyperplane. If the train-
ing examples are not linearly separable, the perceptron
learning algorithm is not guaranteed to converge. Lin-
ear classifiers, like single-unit perceptrons, are only able
because they can only implement a simple decision sur-
(1)
face (single hyperplane) [33]. The same is true for single-
E is
This makes these linear neural networks unable to learn,
for example, the XOR function [33]. Nevertheless a prob-
lem that is thought to be highly complex may still be solved
as well by a linear network as by a more powerful (non-lin-
ear) neural network.
Single-Layer Neural Networks
In general, the state of a neuron is represented by its ac-
tivation value. An activation or transfer function f calcu-
lates the activation value of a unit from the weighted sum s
of its inputs. In case of the perceptron f is called step or
threshold function, with the activation value being 1 if the
network sum is greater than a constant T, and 0 otherwise
(see Eq. (1)). Another common form of non-linear activa-
tion function is the logistic or sigmoid function:
1
f (s) D : (4)
1 C es
This enables a neural network to compute a continuous
output between 0 and 1 instead of a step function. With
1818 D Data-Mining and Knowledge Discovery, Neural Networks in
this choice, a single-layer network is identical to a logistic
regression model. If the activation functions is linear, i. e.,
the identity, then this is just a multiple linear regression
and the output is proportional to the total weighted sum s.
Multi-Layer Neural Networks
The limitation that non-linearly separable functions can-
not be represented by a single-layer network with fixed
weights can be overcome by adding more layers. A multi-
layer network is a feedforward network with two or more
layers of computational units, interconnected such that the
neurons’ outputs of one layer serve as inputs only to neu-
rons of the directly subsequent layer (see Fig. 2). The input
layer is not considered a real layer with processing neu-
rons.
The number of units in the input layer is determined
by the problem, i. e., the dimension of the input data space.
The number of output units also depends on the output
encoding (see Subsect.“ Application Issues”).
By using hidden layers, the partitioning of the data
space can be more effective. In principle, each hidden unit
adds one hyperplane to divide the space and discriminate
the solution. Only if the outputs of at least two neurons
are combined in a third neuron, the XOR problem is solv-
able. Important issues in multi-layer NN design are, thus,
the specification of the number of hidden layers and the
number of units in these layers (see also Subsect. “Appli-
cation Issues”). Both numbers determine the complexity
of functions that can be modeled. There is no theoretical
limitation on the number of hidden layers, but usually one
or two are used.
The universal approximation theorem for neural net-
works states that any continuous function that maps in-
tervals of real numbers to an output interval of real num-
Data-Mining and Knowledge Discovery, Neural Networks in, Fig-
ure 2
Principle structure of multi-layer feedforward neural network
bers can be approximated arbitrarily closely by a multi-
layer neural network with only one hidden layer and cer-
tain types of non-linear activation functions. This gives,
however, no indication about how fast or likely a solution
is found. Networks with two hidden layers may work bet-
ter for some problems. However, more than two hidden
layers usually provide only marginal benefit compared to
the significant increase in training time.
Any multi-layer network with fixed weights and lin-
ear activation function is equivalent to a single-layer (lin-
ear) network: In the case of a two-layer linear system, for
instance, let all input vectors to the first layer form ma-
trix X and W 1 and W 2 be the weight matrices of the two
processing layers. Then the output Y1 D W1  X of the first
layer is input to the second layer, which produces output
Y2 D W2  (W1  X) D (W2  W1 )  X. This is equivalent to
a single-layer network with weight matrix W D W2  W1 .
Only a multi-layer network that is non-linear can pro-
vide more computational power. In many applications
these networks use a sigmoid function as non-linear ac-
tivation function. This is the case at least for the hidden
units. For the output layer, the sigmoid activation func-
tion is usually applied with classification problems, while
a linear transfer function is applied with regression prob-
lems.
Error Function and Error Surface
The error function derives the overall network error from
the difference of the network’s output yij and target out-
put oij over all examples i and output units j. The most-
common error function is the sum squared error:
1 XX
ED (o i j  y i j )2 : (5)
2
i j
Neural network training performs a search within the
space of solution, i. e., all possible network configurations,
towards a global minimum of the error surface. The global
minimum is the best overall solution with the lowest pos-
sible error.
A helpful concept for understanding NN training is
the error surface: The n weight parameters of the network
model form the n dimensions of the search space. For any
possible state of the network or configuration of weights
the error is plotted in the (n C 1)th dimension. The objec-
tive of training is to find the lowest point on this n-dimen-
sional surface. The error surface is seldom smooth. Indeed,
for most problems, the surface is quite rugged with numer-
ous hills and valleys which may cause the network search
to run into a local minimum, i. e., a Suboptimum solution.
 Data-Mining and Knowledge Discovery, Neural Networks in
The speed of learning is the rate of convergence between
the current solution and the global minimum.
In a linear network with a sum-squared error func-
tion, the error surface is an multi-dimensional parabola,
i. e., has only one minimum. In general, it is not possi-
ble to analytically determine where the global minimum of
the error surface is. Training is essentially an exploration
of the error surface. Because of the probabilistic and often
highly non-linear modeling by neural networks, we cannot
be sure that the error could not be lower still, i. e., that the
minimum we found is the absolute one. Since the shape
of the error space cannot be known a priori, neural net-
work analysis requires a number of independent runs to
determine the best solution. When different initial values
for the weights are selected, different network models will
be derived.
From an initially random configuration of the net-
work, i. e., a random point on the error surface, the train-
ing algorithm starts seeking for the global minimum. Small
random values are typically used to initialize the network
weights. Although neural networks resulting from differ-
ent initial weights may have very different parameter set-
tings, their prediction errors usually do not vary dramat-
ically. Training is stopped when a maximum number of
epochs has expired or when the network error does not
improve any further.
Backpropagation
The best known and most popular training algorithm for
multi-layer networks is backpropagation, short for back-
wards error propagation and also referred to as the gener-
alized delta rule [46]. The algorithm involves two phases:
Forward pass. During the first phase, the free parame-
ters (weights) of the network are fixed. An example pattern
is presented to the network and the input signals are prop-
agated through the network layers to calculate the network
w i j D ˛ı j y i :
output at the output unit(s).
Backward pass. During the second phase, the model
parameters are adjusted. The error signals at the output
units, i. e., the differences between calculated and expected
outputs, are propagated back through the network layers.
In doing so, the error at each processing unit is calculated
and used to make adjustments to its connecting weights
such that the overall error of the network is reduced by
some small amount.
After iteratively repeating both phases for a sufficiently
large number of training cycles (epochs) the network will
converge to a state where its output error is small enough.
The backpropagation rule involves the repeated use of the
chain rule, saying that the output error of a neuron can be
D 1819
ascribed partly to errors in the weights of its direct inputs
and partly to errors in the outputs of higher-level (hidden)
nodes [46].
Moreover, backpropagation learning may happen in
two different modes. In sequential mode or online mode
weight adjustments are made example by example, i. e.,
each time an example pattern has been presented to the
network. The batch mode or offline mode adjustments are
made epoch by epoch, i. e., only after all example patterns
have been presented. Theoretically, the backpropagation
algorithm performs gradient descent on the total error
only if the weights are updated epoch-wise. There are em-
pirical indications, however, that a pattern-wise update re-
sults in faster convergence. The training examples should
be presented in random order. Then the precision of pre-
dictions will be more similar over all inputs.
Backpropagation learning requires a differentiable ac-
tivation function. Besides adding non-linearity to multi-
layer networks, the sigmoid activation function (see
Eq. (4)) is often used in backpropagation networks because
it has a continuous derivative that can be calculated easily:
f 0 (s) D f (s)(1  f (s)) : (6)
We further assume that there is only one hidden layer
in order to keep notations and equations clear. A gener-
alization to networks with more than one hidden layer is
straightforward.
The backpropagation rule is a generalization of the
delta learning rule (see Eq. (3)) to multi-layer networks
with non-linear activation function. For an input vector
xE the output y D f (s) is calculated at each output neuron
of the network and compared with the desired target out-
put o, resulting in an error ı. Each weight is adjusted pro-
portionally to its effect on the error. The weight of a con-
nection between a unit i and a unit j is updated depending
on the output of i (as input to j) and the error signal at j:
(7)
For an output node j the error signal (error surface gra-
dient) is given by:
ı j D (o j  y j ) f 0 (s j ) D (o j  y j )y j (1  y j ) : (8)
If the error is zero, no changes are made to the con-
nection weight. The larger the absolute error, the more the
responsible weight is changed, while the sign of the error
determines the direction of change.
For a hidden neuron j the error signal is calculated re-
cursively using the signals of all directly connected output
neurons k.
X X
ı j D f 0 (s j ) ı k w jk D y j (1  y j ) ı k w jk : (9)
k k
1820 D Data-Mining and Knowledge Discovery, Neural Networks in
The partial derivative of the error function with respect
to the network weights can be calculated purely locally,
such that each neuron needs information only from neu-
rons directly connected to it. A theoretical foundation of
the backpropagation algorithm can be found in [31].
The backpropagation algorithm performs a gradient
descent by calculating the gradient vector of the error sur-
face at the current search point. This vector points into the
direction of the steepest descent. Moving in this direction
will decrease the error and will eventually find a new (lo-
cal) minimum, provided that the step size is adapted ap-
propriately. Small steps slow down learning speed, i. e., re-
quire a larger number of iterations. Large steps may con-
verge faster, but may also overstep the solution or make
the algorithm oscillate around a minimum without con-
vergence of the weights. Therefore, the step size is made
proportional to the slope ı, i. e., is reduced when the search
point approaches a minimum, and to the learning rate ˛.
The constant ˛ allows one to control the size of the gra-
dient descent step and is usually set to be between 0.1 and
0.5. For practical purposes, it is recommended to choose
the learning rate as large as possible without leading to os-
cillation.
Momentum
One possibility to avoid oscillation and to achieve faster
convergence is in the addition of a momentum term that is
proportional to the previous weight change:
w i j (t C 1) D ˛ı j y i C ˇ
w i j (t) : (10) The size of the weight step
w i j taken by standard
The algorithm increases learning speed step size if it
has taken several steps in the same direction. This gives it
the ability to overcome obstacles in the error surface, e. g.,
to avoid and escape from local minima, and to move faster
over larger plateaus.
Finding the optimum learning rate ˛ and momentum
scale parameter ˇ, i. e., the best trade-off between longer
training time and instability, can be difficult and might re-
quire many experiments. Global or local adaptation tech-
niques use, for instance, the partial derivative to automati-
cally adapt the learning rate. Examples here are the Delta-
Bar-Delta rule [18] and the SuperSAB algorithm [51].
i j (t  1) if d i j (t  1)  d i j (t) > 0
i j (t) D 
i j (t  1) if d i j (t  1)  d i j (t) < 0
Other Learning Rules
i j (t  1) otherwise
The backpropagation learning algorithm is computation-
ally efficient in that its time complexity is linear in the
number of weight parameters. Its learning speed is com-
paratively low, however, on the basis of epochs. This may
result in long training times, especially for difficult and
complex problems requiring larger networks or larger
amounts of training data.
Another major limitation is that backpropagation does
not always converge. Still, it is a widely used algorithm
and has its advantages: It is relatively easy to apply and
to configure and provides a quick, though not absolutely
perfect solution. Its usually pattern-wise error adjustment
is hardly affected by data that contains a larger number of
redundant examples. Standard backpropagation also gen-
eralizes equally well on small data sets as more advanced
algorithms, e. g., if there is insufficient information avail-
able to find a more precise solution.
There are many variations of the backpropaga-
tion algorithm, like resilient propagation (Rprop) [42],
quick propagation (Quickprop) [13], conjugate gradi-
ent descent [6], Levenberg–Marquardt [16], Delta-Bar-
Delta [18], to mention the most popular. All these second-
order algorithms are designed to deal with some of the lim-
itations on the standard approach. Some work substan-
tially faster in many problem domains, but require more
control parameters than backpropagation, which makes
them more difficult to use.
Resilient Propagation
Resilient propagation (Rprop) as proposed in [42,43] is
a variant of standard backpropagation with very robust
control parameters that are easy to adjust. The algorithm
converges faster than the standard algorithm without be-
ing less accurate.
backpropagation not only depends on the learning rate ˛,
but also on the size of the partial derivative (see Eq. (7)).
This may have an unpredictable influence during training
that is difficult to control. Therefore, Rprop uses only the
sign of derivative to adjust the weights. It necessarily re-
quires learning by epoch, i. e., all adjustments take place
after each epoch only.
One iteration of the Rprop algorithm involves two
steps, the adjustment of the step size and the update of the
weights. The amount of weight change is found by the fol-
lowing update rule:
8 C
< 
:
(11)
with 0 <  < 1 < C and derivative d i j D ı j y i . Every
time t the derivative term changes its sign, indicating that
the last update (at time t  1) was too large and the algo-
 Data-Mining and Knowledge Discovery, Neural Networks in
rithm has jumped over a local minimum, the update value
(step size)
i j (t  1) is decreased by a constant factor  . in time.
The rule for updating the weights is straightforward:
w i j (t C 1) D w i j (t) C
w i j (t) (12) the probability of the case to fall into each class. The vari-
8 ous output classes of a problem are normally represented
< 
i j (t) if d i j (t) > 0 in neural networks using one of two techniques, including
w i j (t) D C
i j (t) if d i j (t) < 0 : (13) binary encoding and one-out-of-n encoding.
:
0 otherwise
One advantage of the Rprop algorithm, compared to,
for example, Quickprop [13], is its small set of parame-
ters that hardly requires adaptation. Standard values for
decrease factor  and increase factor C are 0.5 and 1.2,
respectively. To avoid too large or too small weight val-
ues, this is bounded above by
max and bounded below by
min , set by default to 50 and 106 . The same initial value
0 D 0:1 is recommended for all
ij . While the choice of
the parameter settings is not critical, for most problems
no other choice is needed to obtain the optimal or at least
a nearly optimal solution.
Application Issues
The architecture of a neural network, i.e, the number of
(hidden) neurons and layers, is an important decision. If
a neural network is highly redundant and overparameter-
ized, it might adapt too much to the data. Thus, there is
a trade-off between reducing bias (fitting the training data)
and reducing variance (fitting unknown data). The most-
common procedure is to select a network structure that
has more than enough parameters and neurons and then
to avoid overfitting only over the training algorithm (see
Subsect. “Overtraining and Generalization”).
There is no general best network structure for a par-
ticular type of application. There are only general rules
for selecting the network architecture: (1) The more com-
plex the relationships between input and output data are,
the higher the number of hidden units should be selected.
(2) If the modeled process is separable into multiple stages,
more than one hidden layer may be beneficial. (3) An up-
per bound for the total number of hidden units may be set
by the number of data examples divided by the number of
input and output units and multiplied by a scaling factor.
A simpler rule is to start with one hidden layer and half as
many hidden units as there are input and output units.
One would expect that for a given data set there would
be an optimal network size, lying between a minimum of
one hidden neuron (high bias, low variance) and a very
large number of neurons (low bias, high variance). While
this is true for some data sets, in many cases increasing the
number of hidden nodes continues to improve prediction
D 1821
accuracy, as long as cross validation is used to stop training
For classification problems, the neural network assigns
to each input case a class label or, more generally, estimates
A binary encoding is only possible for two-class prob-
lems. A single unit calculates class 1 if its output is above
the acceptance threshold. If the output is below the rejec-
tion threshold, class 0 is predicted. Otherwise, the output
class is undecided. Should the network output be always
defined, both threshold values must be equal (e. g. 0.5).
In one-out-of-n encoding one unit is allocated for each
class. A class is selected if the corresponding output is
above the acceptance threshold and all the other outputs
are below the rejection threshold. If this condition is not
met, the class is undecided. Alternatively, instead of using
a threshold, a winner-takes-all decision may be made such
that the unit with the highest output gives the class.
For regression problems, the objective is to estimate
the value of a continuous output variable, given the input
variables. Particularly important issues in regression are
output scaling and interpolation. The most-common NN
architectures produce outputs in a limited range. Scaling
algorithms may be applied to the training data to ensure
that the target outputs are in the same range. Constrain-
ing the network’s outputs limits its generalization perfor-
mance. To overcome this, a linear activation function may
be used for the output units. Then there is often no need
for output scaling at all, since the units can in principle
calculate any value.
Other Neural Network Architectures
This section summarizes some alternative NN architec-
tures that are variants or extensions of multi-layer feed-
forward networks.
Cascade Correlation Networks
Cascade-correlation is a neural network architecture with
variable size and topology [14]. The initial network has
no hidden layer and grows during training by adding new
hidden units one at a time. In doing so, a near minimal
network topology is built. In the cascade architecture the
outputs from all existing hidden neurons in the network
are fed into a new neuron. In addition, all neurons – in-
cluding the output neurons – receive all input values.
For each new hidden unit, the learning algorithm tries
to maximize the correlation between this unit’s output and
1822 D Data-Mining and Knowledge Discovery, Neural Networks in
the overall network error using an ordinary learning al-
gorithm, like, e. g., backpropagation. After that the input-
side weighs of the new neuron are frozen. Thus, it does not
change anymore and becomes a permanent feature detec-
tor.
Cascade correlation networks have several advantages
over multi-layer perceptrons: (1) Training time is much
shorter already because the network size is relatively small.
(2) They require only little or no adjustment of parame-
ters, especially not in terms of the number of hidden neu-
rons to use. (3) They are more robust and training is less
likely to become stuck in local minima.
Recurrent Networks
A network architecture with cycles is adopted by recur-
rent or feedback neural networks such that outputs of some
neurons are fed back as extra inputs. Because past outputs
are used to calculate future outputs, the network is said to
“remember” its previous state. Recurrent networks are de-
signed to process sequential information, like time series
data. Processing depends on the state of the network at the
last time step. Consequently, the response to the current
input depends on previous inputs.
Two similar types of recurrent network are extensions
of the multi-layer perceptron: Jordan networks [19] feed
back all network outputs into the input layer; Elman net-
works [12] feed back from the hidden units. State or con-
text units are added to the input layer for the feedback con-
nections which all have constant weight one. At each time
step t, an input vector is propagated in a standard feedfor-
ward fashion, and then a learning rule (usually backprop-
agation) is applied. The extra units always maintain a copy
of the previous outputs at time step t  1.
Radial Basis Function Networks
Radial basic function (RBF) networks [8,34,37,38] are an-
other popular variant of two-layer feedforward neural net-
works which uses radial basis functions as activation func-
tions. The idea behind radial basis functions is to approx-
imate the unknown function f (E x ) by a weighted sum of
non-linear basis functions , which are often Gaussian
functions with a certain standard deviation .
X clustering that implements a mapping of the high-dimen-
x) D
f (E x  cEi jj)
w i (jjE (14)
i
The basis functions operate on the Euclidean distance
between n-dimensional input vector xE and center vector
cEi . Once the center vectors cEi are fixed, the weight coeffi-
cients wi are found by simple linear regression.
The architecture of RBF networks is fixed to two
processing layers. Each unit in the hidden layer repre-
sents a center vector and a basis function which realizes
a non-linear transformation of the inputs. Each output
unit calculates a weighted sum (linear combination) of
the non-linear outputs from the hidden layer. Only the
connections between hidden layer and output layer are
weighted.
The use of a linear output layer in RBF networks is
motivated by Cover’s theorem on the separability of pat-
terns. The theorem states that if the transformation from
the data (input) space to the feature (hidden) space is non-
linear and the dimensionality of the feature space is rela-
tively high compared to that of the data space, then there
is a high likelihood that a non-separable pattern classifica-
tion task in the input space is transformed into a linearly
separable one in the feature space.
The center vectors are selected from the training data,
either randomly or uniformly distributed in the input
space. In principle, as many centers (and hidden units)
may be used as there are data examples. Another method
is to group the data in space using, for example, k-means
clustering, and select the center vectors close to the cluster
centers.
RBF learning is considered a curve-fitting problem in
high-dimensional space, i. e., approximates a surface with
the basis functions that fits and interpolates the training
data points best. The basis functions are well-suited to
online learning applications, like adaptive process con-
trol. Adapting the network to new data and changing data
statistics only requires a retraining by linear regression
which is fast. RBF networks are more local approximators
than multi-layer perceptrons. New training data from one
region of the input space have less effect on the learned
model and its predictions in other regions.
Self-organizing Maps
A self-organizing map (SOM) or Kohonen map [22,23] ap-
plies an unsupervised and competitive learning scheme.
That means that the class labels of the data vectors are
unknown or not used for training and that each neuron
improves through competition with other neurons. It is
a non-deterministic machine-learning approach to data
sional input data into a low-dimensional feature space.
In doing so, SOMs filter and compress information while
preserving the most relevant features of a data set. Com-
plex, non-linear relationships and dependencies are re-
vealed between data vectors and between clusters and are
transformed into simple geometric distances. Such an ab-
 Data-Mining and Knowledge Discovery, Neural Networks in
straction facilitates both visualization and interpretation
of the clustering result.
Typically, the units of a SOM network are arranged
in a two-dimensional regular grid, the topological feature
map, which defines a two-dimensional Euclidean distance
between units. Each unit is assigned a center vector from
the n-dimensional data space and represents a certain clus-
ter.
Algorithm 1 describes the basic principle behind SOM
training. Starting with an initially random set of center
vectors, the algorithm iteratively adjusts them to reflect
the clustering of the training data. In doing so, the two-
dimensional order of the SOM units is imposed on the in-
put vectors such that more similar clusters (center vectors)
in the input space are closer to each other on the two-di-
mensional grid structure than more different clusters. One
can think of the topological map to be folded and distorted
into the n-dimensional input space, so as to preserve as
much as possible the original structure of the data.
Algorithm 1 (Self-organizing Map)
1. Initialize the n-dimensional center vector cEi 2 Rn of
each cluster randomly.
2. For each data point pE 2 Rn find the nearest center vec-
tor cEw (called the “winner”) in n-dimensional space ac-
cording to a distance metric d.
3. Move cEw and all centers cEi within its local neighbor-
hood closer to pE :
cEi (t C 1) D cEi (t) C ˛(t)  hr(t)  ( pE  cEi (t))
with learning rate 0 < ˛ < 1 and neighborhood func-
tion h depending on a neighborhood radius r.
4. ˛(t C 1) D ˛(t) 
˛ where
˛ D ˛0 /tmax
r(t C 1) D r(t) 
r where
r D r0 /tmax native to classical statistical methods and are frequently
5. Repeat steps 2.–4. for each epoch t D 1; ::; tmax .
6. Assign each data point to the cluster with the nearest
center vector.
Each iteration involves randomly selecting a data point
pE and moving the closest center vector a bit in the di-
rection of pE. Only distance metric d (usually Euclidean)
defined on the data space influences the selection of
the closest cluster. The adjustment of centers is applied
not just to the winning neuron, but to all the neurons
of its neighborhood. The neighborhood function is often
Gaussian. A simple definition calculates h D 1 if the Eu-
clidean distance between the grid coordinates of the win-
ning cluster w and a cluster i is below a radius r, i. e.,
jj(xw ; yw )  (x i ; y i )jj < r, and h D 0 otherwise.
D 1823
Both learning rate ˛ and neighborhood radius r mono-
tonically decrease over time. Initially quite large areas of
the network are affected by the neighborhood update,
leading to a rather rough topological order. As epochs
pass, less neurons are altered with lower intensity and finer
distinctions are drawn within areas of the map.
Unlike hierarchical clustering [52] and k-means clus-
tering [29], which are both deterministic – apart from the
randomized initialization in k-means clustering – and op-
erate only locally, SOMs are less likely to become stuck in
local minima and have a higher robustness and accuracy.
Once the network has been trained to recognize struc-
tures in the data, it can be used as a visualization tool and
for exploratory data analysis. The example map in Fig. 3
shows a clustering of time series of gene expression val-
ues [48]. Clearly higher similarities between neighboring
clusters are revealed when comparing the mean vectors.
If neurons in the feature map can be labeled, i. e.,
if common meaning can be derived from the vectors in
a clusters, the network becomes capable of classification.
If the winning neuron of an unknown input case has not
been assigned a class label, labels of clusters in close or di-
rect neighborhood may be considered. Ideally, higher sim-
ilarities between neighboring data clusters are reflected in
similar class labels. Alternatively, the network output is
undefined in this case. SOM classifiers also make use of
the distance of the winning neuron from the input case.
If this distance exceeds a certain maximum threshold, the
SOM is regarded as undecided. In this way, a SOM can be
used for detecting novel data classes.
Future Directions
To date, neural networks are widely accepted as an alter-
used in medicine [4,5,11,25,27,44,45] with many applica-
tions related to cancer research [10,26,35,36,49]. In the
first place, these comprise diagnostics and prognosis (i. e.
classification) tasks, but also image analysis and drug de-
sign. Cancer prediction is often based on clustering of
gene expression data [15,50] or microRNA expression
profiles [28] which may involve both self-organizing maps
and multi-layer feedforward neural networks.
Another broad application field of neural networks
today is bioinformatics and, in particular, the analysis
and classification of gene and protein sequences [3,20,56].
A well-known successful example is protein (secondary)
structure prediction from sequence [39,40].
Even though the NN technology is clearly established
today, the current period is rather characterized by stag-
nation. This is partly because of a redirection of research
1824 D Data-Mining and Knowledge Discovery, Neural Networks in

Data-Mining and Knowledge Discovery, Neural Networks in, Figure 3

6 x 6 SOM example clustering of gene expression data (time series over 24 time points) [48]. Mean expression vector plotted for each
cluster. Cluster sizes indicate the number of vectors (genes) in each cluster

to newer and often – but not generally – more power-
ful paradigms, like the popular support vector machines
(SVMs) [9,47], or to more open and flexible methods,
like genetic programming (GP) [7,24]. In many applica-
tions, for example, in bioinformatics, SVMs have already
replaced conventional neural networks as the state-of-the-
art black-box classifier.
Bibliography
Primary Literature
1. Ackley DH, Hinton GF, Sejnowski TJ (1985) A learning algo-
rithm for Boltzman machines. Cogn Sci 9:147–169
2. Anderson JA et al (1977) Distinctive features, categorical per-
ception, and probability learning: Some applications of a neu-
ral model. Psychol Rev 84:413–451
3. Baldi P, Brunak S (2001) Bioinformatics: The machine learning
approach. MIT Press, Cambridge
4. Baxt WG (1995) Applications of artificial neural networks to
clinical medicine. Lancet 346:1135–1138
5. Begg R, Kamruzzaman J, Sarkar R (2006) Neural networks in
healthcare: Potential and challenges. Idea Group Publishing,
Hershey
6. Bishop CM (1995) Neural networks for pattern recognition. Ox-
ford University Press, London
7. Brameier M, Banzhaf W (2007) Linear genetic programming.
Springer, New York
8. Broomhead DS, Lowe D (1988) Multivariable functional inter-
polation and adaptive networks. Complex Syst 2:321–355
9. Cristianini N (2000) An introduction to support vector ma-
chines and other kernel-based learning methods. Cambridge
University Press, London
10. Dybowski R (2000) Neural computation in medicine: Perspec-
tives and prospects. In: Malmgren H, Borga M, Niklasson L (eds)
Proceedings of the Conference on Artificial Neural Networks in
Medicine and Biology (ANNIMAB). Springer, Berlin, pp 26–36
11. Dybowski R, Gant V (2001) Clinical Applications of Artificial
Neural Networks. Cambridge University Press, London
12. Elman JL (1990) Finding structure in time. Cogn Sci 14:179–211
13. Fahlman SE (1989) Faster learning variations on backpropa-
gation: An empirical study. In: Touretzky DS, Hinton GE, Se-
jnowski TJ (eds) Proceedings of the (1988) Connectionist Mod-
els Summer School. Morgan Kaufmann, San Mateo, pp 38–51
 Data-Mining and Knowledge Discovery, Neural Networks in
14. Fahlman SE, Lebiere C (1990) The cascade-correlation learning
architecture. In: Touretzky DS (ed) Advances in Neural Informa-
tion Processing Systems 2. Morgan Kaufmann, Los Altos
15. Golub TR et al (1999) Molecular classification of cancer: class
discovery and class prediction by gene expression monitoring.
Science 286(5439):531–537
16. Hagan MT, Menhaj M (1994) Training feedforward net-
works with the Marquardt algorithm. IEEE Trans Neural Netw
5(6):989–993
17. Hopfield JJ (1982) Neural networks and physical systems with
emergent collective computational abilities. Proc Natl Acad Sci
79(8):2554–2558
18. Jacobs RA (1988) Increased rates of convergence through
learning rate adaptation. Neural Netw 1:295–307
19. Jordan MI (1986) Attractor dynamics and parallelism in a con-
nectionist sequential machine. In: Proceedings of the Eighth
Annual Conf of the Cogn Sci Society. Lawrence Erlbaum, Hills-
dale, pp 531–546
20. Keedwell E, Narayanan A (2005) Intelligent bioinformatics: The
application of artificial intelligence techniques to bioinformat-
ics problems. Wiley, New York
21. Kohonen T (1977) Associative Memory: A System-Theoretical
Approach. Springer, Berlin
22. Kohonen T (1982) Self-organized formation of topologically
correct feature maps. Biol Cybern 43:59–69
23. Kohonen T (1995) Self-organizing maps. Springer, Berlin
24. Koza JR (1992) Genetic programming: On the programming
of computer programs by natural selection. MIT Press, Cam-
bridge
25. Lisboa PJG (2002) A review of evidence of health benefit from
artificial neural networks in medical intervention. Neural Netw
15:11–39
26. Lisboa PJG, Taktak AFG (2006) The use of artificial neural net-
works in decision support in cancer: a systematic review. Neu-
ral Netw 19(4):408–415
27. Lisboa PJG, Ifeachor EC, Szczepaniak PS (2001) Artificial neural
networks in biomedicine. Springer, Berlin
28. Lu J et al (2005) MicroRNA expression profiles classify human
cancers. Nature 435:834–838
29. MacQueen J (1967) Some methods for classification and anal-
ysis of multivariate observations. In: Proceedings of the 5th
Berkeley Symposium on Mathematical Statistics and Probabil-
ity, vol 1. University of California Press, Berkeley, pp 281–297
30. McClelland JL, Rumelhart DE (1986) Parallel distributed pro-
cessing: Explorations in the microstructure of cognition. MIT
Press, Cambridge
31. McClelland J, Rumelhart D (1988) Explorations in parallel dis-
tributed processing. MIT Press, Cambridge
32. McCulloch WS, Pitts W (1943) A logical calculus of the ideas im-
manent in nervous activity. Bull Math Biophys 5:115–133
33. Minsky ML, Papert SA (1969/1988) Perceptrons. MIT Press,
Cambridge
34. Moody J, Darken CJ (1989) Fast learning in networks of locally-
tuned processing units. Neural Comput 1:281–294
35. Naguib RN, Sherbet GV (1997) Artificial neural networks in can-
cer research. Pathobiology 65(3):129–139
36. Naguib RNG, Sherbet GV (2001) Artificial neural networks in
cancer diagnosis, prognosis, and patient management. CRC
Press, Boca Raton
37. Poggio T, Girosi F (1990) Regularization algorithms for learning
D 1825
that are equivalent to multi-layer networks. Science 247:978–
982
38. Poggio T, Girosi F (1990) Networks for approximation and
learning. Proc IEEE 78:1481–1497
39. Quian N, Sejnowski TJ (1988) Predicting the secondary struc-
ture of globular proteins using neural network models. J Mol
Biol 202:865–884
40. Rost B (2001) Review: Protein secondary structure prediction
continues to rise. J Struct Biol 134:204–218
41. Rosenblatt F (1958) The perceptron: A probabilistic model for
information storage and organization in the brain. Psychol Rev
65(6):386–408
42. Riedmiller M, Braun H (1992) Rprop – a fast adaptive learning
algorithm. In: Proceedings of the International Symposium on
Computer and Information Science VII
43. Riedmiller M, Braun H (1993) A direct adaptive method for
faster backpropagation learning: The Rprop algorithm. In: Pro-
ceedings of the IEEE International Conference on Neural Net-
works. IEEE Press, Piscataway, pp 586–591
44. Ripley BD, Ripley RM (2001) Neural Networks as statistical
methods in survival analysis. In: Dybowski R, Gant V (eds) Clini-
cal Applications of Artificial Neural Networks. Cambridge Univ
Press, London
45. Robert C et al (2004) Bibliometric overview of the utilization of
artificial neural networks in medicine and biology. Scientomet-
rics 59:117–130
46. Rumelhart DE, Hinton GE, Williams RJ (1986) Learning repre-
sentations by backpropagating errors. Nature 323:533–536
47. Schlökopf B, Smola AJ (2001) Learning with kernels: Support
vector machines, regularization, optimization, and beyond.
MIT Press, Cambridge
48. Spellman PT et al (1998) Comprehensive identification of cell
cycle-regulated genes of the yeast Saccharomyces cerevisiae
by microarray hybridization. Mol Biol Cell 9(12):3273–97
49. Taktak AFG, Fisher AC (2007) Outcome prediction in cancer. El-
sevier Science, London
50. Tamayo P et al (1999) Interpreting patterns of gene expres-
sion with self-organizing maps: Methods and application to
hematopoietic differentiation. PNAS 96(6):2907–2912
51. Tollenaere T (1990) SuperSAB: Fast adaptive backpropagation
with good scaling properties. Neural Netw 3:561–573
52. Ward JH (1963) Hierarchical grouping to optimize an objective
function. J Am Stat Assoc 58(301):236–244
53. Werbos PJ (1974) Beyond regression: New tools for prediction
and analysis in the behavioral science. Ph D Thesis, Harvard
University
54. Werbos PJ (1994) The roots of backpropagation. Wiley, New
York
55. Widrow B, Hoff ME (1960) Adaptive switching circuits. In:
IRE WESCON Convention Record, Institute of Radio Engineers
(now IEEE), vol 4. pp 96–104
56. Wu CH, McLarty JW (2000) Neural networks and genome infor-
matics. Elsevier Science, Amsterdam
Books and Reviews
Abdi H (1994) A neural network primer. J Biol Syst 2:247–281
Bishop CM (2008) Pattern recognition and machine learning.
Springer, Berlin
Fausett L (1994) Fundamentals of neural networks: Architectures,
algorithms, and applications. Prentice Hall, New York
1826 D Decision Trees
Freeman JA, Skapura DM (1991) Neural networks: Algorithms, ap-
plications, and programming techniques. Addison, Reading
Gurney K (1997) An Introduction to neural networks. Routledge,
London
Hastie T, Tibshirani R, Friedman JH (2003) The elements of statistical
learning. Springer, Berlin
Haykin S (1999) Neural networks: A comprehensive foundation.
Prentice Hall, New York
Hertz J, Krogh A, Palmer R (1991) Introduction to the theory of neu-
ral computation. Addison, Redwood City
Kröse B, van der Smagt P (1996) An introduction to neural net-
works. University of Amsterdam, Amsterdam
Masters T (1993) Practical neural network recipes in C++. Academic
Press, San Diego
Masters T (1995) Advanced algorithms for neural networks: A C++
sourcebook. Wiley, New York
Parks R, Levine D, Long D (1998) Fundamentals of neural network
modeling. MIT Press, Cambridge
Patterson D (1996) Artif neural networks. Prentice Hall, New York
Peretto P (1992) An introduction to the modeling of neural net-
works. Cambridge University Press, London
Ripley BD (1996/2007) Pattern recognition and neural networks.
Cambridge University Press, London
Smith M (1993) Neural networks for statistical modeling. Van Nos-
trand Reinhold, New York
Wasserman PD (1989) Neural Computing: Theory and Practice. Van
Nostrand Reinhold, New York
Wasserman PD (1993) Advanced methods in neural computing.
Van Nostrand Reinhold, New York
De Veaux RD, Ungar LH (1997) A brief introduction to neural net-
works. Technical Report, Williams College, University of Penn-
sylvania
Hinton GE (1992) How neural networks learn from experience. Sci
Am 267:144–151
Lippman RP (1987) An introduction to computing neural networks.
IEEE ASSP Mag 4(2):4–22
Reilly D, Cooper L (1990) An overview of neural networks: Early
models to real world systems. Neural Electron Netw 2:229–250
Decision Trees
VILI PODGORELEC, MILAN Z ORMAN
University of Maribor, Maribor, Slovenia
Article Outline
Glossary
Definition of the Subject
Introduction
The Basics of Decision Trees
Induction of Decision Trees
Evaluation of Quality
Applications and Available Software
Future Directions
Bibliography
Glossary
Accuracy The most important quality measure of an in-
duced decision tree classifier. The most general is the
overall accuracy, defined as a percentage of correctly
classified instances from all instances (correctly classi-
fied and not correctly classified). The accuracy is usu-
ally measured both for the training set and the testing
set.
Attribute A feature that describes an aspect of an object
(both training and testing) used for a decision tree. An
object is typically represented as a vector of attribute
values. There are two types of attributes: continuous
attributes whose domain is numerical, and discrete at-
tributes whose domain is a set of predetermined val-
ues. There is one distinguished attribute called deci-
sion class (a dependent attribute). The remaining at-
tributes (the independent attributes) are used to deter-
mine the value of the decision class.
Attribute node Also called a test node. It is an internal
node in the decision tree model that is used to deter-
mine a branch from this node based on the value of the
corresponding attribute of an object being classified.
Classification A process of mapping instances (i. e. train-
ing or testing objects) represented by attribute-value
vectors to decision classes. If the predicted decision
class of an object is equal to the actual decision class
of the object, then the classification of the object is ac-
curate. The aim of classification methods is to classify
objects with the highest possible accuracy.
Classifier A model built upon the training set used for
classification. The input to a classifier is an object
(a vector of known values of the attributes) and the
output of the classifier is the predicted decision class
for this object.
Decision node A leaf in a decision tree model (also called
a decision) containing one of the possible decision
classes. It is used to determine the predicted decision
class of an object being classified that arrives to the leaf
on its path through the decision tree model.
Instance Also called an object (training and testing), rep-
resented by attribute-value vectors. Instances are used
to describe the domain data.
Induction Inductive inference is the process of moving
from concrete examples to general models, where the
goal is to learn how to classify objects by analyzing
a set of instances (already solved cases) whose classes
are known. Instances are typically represented as at-
tribute-value vectors. Learning input consists of a set
of such vectors, each belonging to a known class, and
the output consists of a mapping from attribute val-