Generalization Capacity in Artificial Neural Networks

- the ballistic trajectory learning case -

M. Roisenberg1, J. M. Barreto2, F. M. Azevedo2

1Instituto de Informática, Departamento de Informática Aplicada,

Universidade Federal do Rio Grande do Sul,
Porto Alegre, Brasil
e-mail: roisenb@inf.ufrgs.br
(Atualmente em programa de doutorado no endereço 2)

2Grupo de Pesquisas em Engenharia Biomédica, Departamento de Engenharia Elétrica,

Universidade Federal de Santa Catarina,
Florianópolis, Brasil
e-mail: barreto@gpeb.ufsc.br, azevedo@gpeb.ufsc.br

Abstract
This paper presents de use of an Artificial Neural Network (ANN) in learning the features of a
dynamic system, in such case, the development of an ANN as the controller for a device designed to
launch objects under ballistic movement. We make considerations about the generalization capacity
both for live beings as for the network. We also present the network topology and configuration, the
learning technique and the precision obtained. At last, this work shows the importance of the choice of
the activation function in learning some critical points.

1. Introduction depends on the initial conditions of the

The ballistic movement interception
capacity is pervasive to a lot of life beings.
Observing some situations in our everyday life,
we can imagine that the interception of objects
under ballistic movement is a trivial action.
Consider for example, when we throw a wood
stick to a dog, or even during football,
basketball or other games. In these situations,
the interception task is always present but we
rarely think about how it's done. The other
problem in achieving that action, resides in the
required response time -- when someone throws
you a ball, you have just a few seconds to
predict its trajectory and to act to intercept it at
the right point. Another great problem is that
the ballistic movement, whose trajectory is
given by the following equations:
x = vo * cos(θ) * t (1)
and
y = vo * sin(θ) * t - g * t2 / 2 (2)
launching, that is, the initial launching angle
and velocity. The combination on both of these
variables is capable to generate trajectories that
fulfill all the state space ℜ 2.
As there are an infinite number of
possible trajectories, it is "not trivial" to train a
neural network to solve this problem (so easily
solved by human beings). We must count on the
neural network generalization capacity. As it's
impossible to train a neural network with all the
possible trajectories set, the neural network
generalization capacity must be used, so that
we train the network with some trajectories and
it can correctly predict all the others.
2. Generalization vs. Specialization
Artificial Neural Networks are capable
to estimate continuous functions through the
observation on how the output data relates with
the process inputs, without mathematically
specifying this relationship. Due to this feature,
the neural networks are becoming largely
utilized by the engineering community in
identification problems and for control of
dynamic systems. Once a neural network has
learned the desired relationship between the
input and output data presented during the
training phase, we hope that it gives right
answers to other problems of the same type, but
it didn't ever seen, in the execution phase,
presenting the expected generalization feature.
One of the most interesting features of
intelligent life beings is the specialization
versus generalization dilemma. In this case,
specialization means to be precise, accurate in
producing the right answers to some given
questions. In the other way, generalization
means the property to give right answers (not
necessarily accurate answers) to questions not
previously seen in the examples.
Some animals are very specialized in
some aspects of their behaviors or anatomy that
help them to survive at their natural
environment, like the Panda Bear that just feeds
some kind of bamboo. By the other side, this
specialization makes an adaptation to other
conditions or the learning of other tasks very
difficult. Animals not so specialized, like
humans, have a higher adaptation and survival
capacity, may be due to their generalization
capacity. In the computational area, traditional
computer programs have a very good
performance in solving algorithmic problems,
but they have much less performance in ill
defined, imprecise, fuzzy problems. On the
other hand, humans are very poor in solving
algorithmic problems, but they can predict the
ballistic trajectory of an object, even if it has
been launched with an angle or velocity never
seen before. We expect that other
computational approaches, like the neural
computing paradigm, have a better performance
in these ill defined and imprecise class of
problems.
3. Implementation
As already said, it's impossible to train
the neural network with all the ballistic
trajectories generated by different launching
velocities and launching angles. So, we must
choose an adequate set of velocities and
launching angles to train the network. Once
trained for these trajectories, we expect that the
neural network could generalize, i.e., to predict
the real trajectory for angles and velocities not
seen in the training examples.
In order to implement the interception
device, we had to study some subjects: the
generalization capacity of neural networks to
solve this problem, how precise are the given
responses, and what the network must learn
(which trajectories must be presented to the
network during the training phase). So, we
implemented two networks and selected two
situations to train the networks. In the first
situation, the direct model of the ballistic
movement was presented to the network, i.e.,
for a given launching angle, velocity and time
after the launching, we want to obtain the
position of the launched object in the plane
(x,y). In the second case, the situation was
inverted and the inverse model was presented to
the network. In this case, the input data to the
network was a target position point and the
output was the launching angle and velocity to
reach this target point.
Suppose a device designed to intercept
objects under ballistic movement, for example a
basket ball player like Michael Jordan. The two
situations previously described take place
almost at the same time. As soon as he ("Air"
Jordan) detects an object under ballistic
movement (the ball), the interception device
tries to predict the trajectory and choose the
interception point where the object will be at
certain time in future. Following this choice, the
interception devices (Michael Jordan) launch
himself in a ballistic trajectory with some
velocity and launching angle, in order to
intercept the object at the previewed point and
then do the points throwing the ball into the
basket (another ballistic movement)
The neural network used to implement
the direct and the inverse model has a feed
forward architecture with one hidden layer. The
learning algorithm used was back propagation.
For the first model the network has three
neurons at the input layer and two neurons at
the output layer, as we can see at figure 1a. The
input neurons receive the considered time
instant, the launching angle and velocity. At the
output layer, we obtain the corresponding
position points (x,y)
For the second case, the two input
neurons of the network receive the target points
coordinates (x,y) and produces at the two
output neurons the launching angle and velocity
to reach the desired point as can be seen at
figure 1b.
Another question that we were faced
during the implementation of the neural
network to run the experiments, became the
choice of the activation function (also known by
threshold, squashing, signal or output function)
be implemented in the neurons. Traditionally, in
many commercial products, a function called
logistic seems to be frequently preferred to be
implemented, due to its simple derivative used
in the back propagation algorithm. However,
the image of this function belongs to the range
[0,1]. So, the network output could only has
positive values in this range. That's OK for
horizontal distances from the launching point (x
coordinate points), but means a hard work
when we have negative height values (y
coordinate points). A more natural choice,
would be to use an activation function that had
image values in the range [-1,1]. This would
allow to train the network with trajectories that
reach negative height values, as when the
launch angle or velocity is small. For this case,
we choose an activation function in the
hyperbolic tangent form.
We made experiments with both the
activation functions for the neural network
neurons. Firstly we implement the logistic
function:
(1 + e-x)-1
and secondly, the hyperbolic tangent function:
(1 - e-x) / (1 + e-x) .
Our experiments confirmed the stated
by Gallant[3]. The learning performance
presented by the network using the hyperbolic
tangent activation function was higher than that
presented when we used the logistic function.
This can be explained because, in the first
moments after the launching, the output values
produced by the network are near zero. The
derivative of the logistic function for this points
is also very small, then the gradient method
used by the back propagation algorithm
presents greater residual error. In using the
hyperbolic tangent activation function, that
don't happen. In such case, the derivative is
bigger just for points near zero.
4. Experiments
In this section we will describe the
experiments we have done to study the
generalization capacity of the network, the
features of the training set, as well as the
influence of the number of neurons in the
hidden layer.
4.1. experiment no 1
In order to analyze the generalization
capacity, the neural network was trained to
learn the ballistic trajectory of an object
launched with one initial velocity (10 m/s) and
two launching angles (30o and 60o). Some
point coordinates (x,y) followed by an object
launched under these two initial conditions,
during 2 seconds, were presented to the network
during the training phase, as can be seen in
figure 2. At the execution phase, we recall from
the trained network the 2 seconds' trajectory for
an object launched with the same initial
velocity, but launching angle varying from 0o
to 90o. We plotted the maximum error between
the real trajectory and that given by the trained
network. These error values can be seen in
figure 3.
We can see that for launching angles
inside the range compound by the angles used
to train the network, the interpolation done by
the network was very good. However, the
extrapolation capacity for angles outside the
trained angles' interval was very poor.
4.2. experiment no 2
In the second experiment we change the
training set. In this case, we want to observe the
interpolation capacity if with train the network
with other two trajectories far away from each
other. Now we trained the network maintaining
the initial velocity but with other launching
angles (15o and 75o). As can be seen in figure
4, the interpolation performance presented by
the network during the execution phase for
launching angles between the trained interval
was not good. As stated previously, this poorer
performance occurs because the "concepts" in
this case the two different trajectories presented
to the network during the training phase are too
"distant" from each other, exceeding the neural
network generalization capacity. In this
experiment we also vary the number of neurons
in the hidden layer, but can't observe any
significant variance in the generalization
capacity. At last we trained the network with an
intermediate angle (15o , 45o and 75o ). The
result can be seen at figure 4.
4.3. experiment no 3
For the third experiment, the inverse
model previously described was implemented.
In this case our goal was to teach to the neural
network some features of the inverse model.
Given some target coordinate point (x,y) to the
network input, we want to get from the network
output the right launching angle and velocity to
reach the point. As there are infinite solutions
to reach the target point, we give as solution to
the network the smaller angle (in 15o interval)
and the corresponding launching velocity. In
this example the number and complexity of the
"concepts" to be taught to the network are much
greater. Moreover, for a lot of points in the
space of coordinates, mainly for those located
near the launching point, we must constantly
vary the solution angle and velocity during the
training phase.
To illustrate the variability of
"concepts" presented to the network, mainly at
target points near the launching point, look at
the following example: In some situations we
must to "increase" the launching angle while
"reducing" the launching velocity to reach a
target point more distant than other relatively
with the launching point (to reach the point
(1.5,2.0) the launching conditions taught to the
network are: θo=60o and vo=8.6m/s, while to
reach a far point (1.5,2.5) the launching angle
grows to θo =75o, but the initial velocity
decreases to vo=7.3m/s). In other situations, the
relation changes (to reach the point (1.5,0.5)
the launching conditions taught to the network
are: θo =30o e vo=6.34m/s, while to reach a
far point (1.5,1.0) the launching angle grows to
θo =45o, and the initial velocity also grows to
vo=6.64m/s).
For more distant points from the
launching point, the variation in the "launching
angle" parameter occurs less frequently, and we
just have to increase the "launching velocity"
parameter to reach target far points. The
"concepts" to be taught to the network are more
"homogeneous".
After a long training phase, the results
presented by the network can be seen in figure
5. There we can see that for target points near
the launching point, the errors are large and the
interpolation capacity presented by the network
are poor. For target points faraway from the
origin, where the relationship "greater distance,
greater launching velocity" is more constant,
the generalization capacity as the precision
obtained by the network are greater.
5. Conclusions
The results obtained through the
experiments leads us to conclude that, in
general, artificial neural networks have a good
generalization capacity, in the sense that they
can produce correct responses (not exactly
precise ones) to questions that does not belong
to the training set. However, we note that this
generalization occurs in the interpolation sense
between two or more previously learned
"concepts". It's important to remark that the
"distance" between the concepts, i.e., how
different the "concepts" are, plays an essential
role in the network generalization capacity.
This shows the great importance of the
selection of the examples in the training set. In
what concerns about the extrapolation capacity
of the neural networks, as for the humans, is a
much more complex task and as the humans,
the neural networks tend to produce wrong
results.
In this work, the concepts of "concept
taught to a neural network" and "distance
between concepts" were presented in an
informal and intuitive way. We intend to
formalize these concepts based in
.mathematically based.. approach in a future
work.
At last, this work tries to show the
importance in choosing the neurons' activation
function in learning performance of some
critical points. Moreover, we presented an
application in which some features of a
dynamic system were learned by a static neural
network.
This fact is very important. In fact,
suppose the control of a plant performed by
conventional frequency response methods. It is
well known that it is impossible to control the
plant with even a non-linear gain in the
feedback loop in several cases, for exemple a
system with two integrators and without zeros.
In this case for any gain the frequency response
will have the -1 point inside the Nyquist loop,
Figure 2. Real and neural network generated
indicating instability of the closed loop system.
trajectory for some launching angles and
In this case it is necessary to use a dynamic
launching velocity of 10 m/s.
system in the controller, in such a way to
reshape the frequency response and to leave the
-1 point outside the Nyquist loop. Or, using a
ANN in this case must include some sort of
dynamics.
In our case, however, we are learning
not the dynamics of the system, but the
trajectories that are functions in a space where
time is not a parameter, but a coordinate
variable, and it is possible to learn functions
with a static system implemented by the ANN.

Time X Position
Angle
Y Position
Velocity
...
Figure 3. Maximum error between the real and
the neural network calculated trajectory for
Figure 1a various launching angles and launching velocity
of 10m/s. The network was trained for 30o and
X Position 60o launching angles.
Angle

Velocity
Y Position
...

Figure 1b

Figure 1. Topology of the networks used to

learn the ballistic movement in an interception
device.
Figure 4. Maximum error between the real and for the MUCOM (Multisensory Control of
the neural network calculated trajectory for Movement, EU project), 1994.
various launching angles and launching velocity
[7] C. C. Klimasaukas, "Neural Networks: an
of 10m/s. The network was firstly trained for
Engineering Perspective," IEEE
15o and 75o launching angles and then for 15o,
Communications Magazine, p50-53, Sept.,
45o and 75o launching angles.
1992.
[8] J. L. Moody, "The Effective Number of
Parameters: An Analysis of Generalization
and Regularization in Nonlinear Learning
Systems." In J. E. Moody and S. J.
Hanson (Eds.), Neural Information
Processing Systems 4 (Morgan Kaufmann,
1992).
[9] J. M. Barreto, F. M. de Azevedo, C. I.
Zanchin, "Neural Network Identification
of Ressonance Frequencies from Noise".
In 9th Brazilian Congress on Automatics,
p840-844, Aug., 1992.
Figure 5. Minimum distance between the given
target point and the closer point reached by an
object launched with the initial conditions
calculated by the network.

References
[1] J. A. Anderson, "Neural-Network Learning
and Mark Twain's Cat," IEEE
Communications Magazine, p16-23,
September, 1992.
[2] D. E. Rummelhart, G. E. Hinton and R. J.
Williams, "Learning Internal
Representations by Error Propagation.") in
D. E. Rummelhart and J. L. McClelland
(Eds.), Parallel Distributed Processing -
Vol. 1, (MIT Press, 1989).
[3] S. I. Gallant, "Neural Network Learning
and Expert Systems," (MIT Press, 1993).
[4] R. Resnick, D. Halliday, "Physics, Part I,"
(Jonh Willey & Sons, 1973).
[5] Z. Schreter, "Connectionism: A Link
Between Psychology and Neuroscience?"
In J Stender, T. Addis (Eds.), Symbols
versus Neurons? (IOS Press, 1990).
[6] J. Barreto, T. Proychev, "Control of the
Standing Position: A Neural Network
Approach," Technical Report, Lab. of
Neurophysiology, Medicine Faculty,
University of Louvain, Brussels, prepared