Proceedings of the Eighth International Conference on Machine Learning and Cybernetics, Baoding, 12-15 July 2009
USE OF NEURAL NETWORKS AS DECISION MAKERS IN STRATEGIC
SITUATIONS
BENOIT COURAUD, PEILIN LIU
Department of Electronic Engineering, Shanghai Jiao Tong University, Shanghai, China
E-MAIL: benoit.couraud@sjtu.edu.cn, liupeilin@sjtu.edu.cn
Abstract:
Intelligence consists of the ability to make right decisions
in a given situation in order to achieve a certain goal. Game
Theory provides mathematical models of real-world situations
for studying intelligent behavior. Most of time, effective
decision-making in strategic situations (such as competitive
situations) requires nonlinear mapping between stimulus and
response. This sort of mapping can be provided by Artificial
Neural Networks. This paper describes the use of a
human-like Artificial Neural Network to find the optimal
strategy in strategic situations without injecting expert
knowledge. In order to train such a Neural Network, an
unsupervised reinforcement-learning rule using
Back-Propagation is introduced. Unlike most of reinforcement
learning systems, this learning rule can operate with
continuous outputs, what makes it worth for a lot of different
applications. Finally, this decision maker is used to find the
optimal strategy in the well-known Iterated Prisoner’s
Dilemma, in order to demonstrate that this human-like
Artificial Neural Networks can be used to design machines
that are also capable of intelligent behavior.
Keywords:
Neural Networks; Artificial Intelligence; Reinforcement
training; Back-Propagation; Game Theory; Iterated
Prisoner’s Dilemma
1. Introduction
The main goal of Artificial Intelligence is to design
machines that are capable of intelligent behavior. This
intelligent behavior is usually described in terms of
stimulus-response: the intelligent agent chooses one
strategy (one action) in response to a particular situation.
This situation can be one of the numerous real-world
situations described by Game Theory.
The fundamentals of game theory were first laid out in
[1]. Game theory attempts to mathematically study optimal
behavior in strategic situations in which one’s success
depends on the choices of the other players. These strategic
situations describe real-world situations such as those that
occur in political science and economics [1], in evolution
978-1-4244-3703-0/09/$25.00 ©2009 IEEE
1280
[2], or in social dilemmas [3]. In these situations, two or
more intelligent agents (players) have the choice between
different strategies that will give them a payoff that also
depends on the other players’ strategies.
One of the most famous and useful game in economics
is the prisoner Dilemma: consider the case where you,
player A, play a game against an opponent, B, with whom
you cannot communicate. You each have two plays: C
(cooperate) or D (defect). If you both play C, you will both
receive 3$. If you play C and they play D, you receive
nothing. Alternatively, if you play D and they play C, you
receive 5$. But if you both play D, you two will receive 1$.
The one shot’s prisoner dilemma (each player just has
one move) has a single dominant strategy: Defect. But in
the case of the Iterated Prisoner Dilemma (IPD), the
optimal strategy seems to be “Tit-for-Tat” (Cooperate on
the first move, and then, for every other moves, mimic the
strategy of the other player at the precedent move), as show
the analysis of Axelrod in [3] and [4].
This paper first introduces a Human-like Artificial
Neural Networks (HLANN) and its own
reinforcement-learning rule using Back-Propagation. Then,
it exhibits one example of its utilization as an intelligent
agent that is able to make the right decision in a strategic
situation such as the one of the Iterated Prisoner Dilemma.
2. Human-like Artificial Neural Network
2.1. Human decisions
In order to design an ANN that is capable of intelligent
behaviors such as those of humans, we first have to study
the process that leads to human decision-making. In a given
situation, humans know the conditions of the situation, the
possible choices they have, how much they like each choice,
and what they want. Let us study a simple situation:
everyday, you know the weather outside (it is a condition),
and you have two choices: stay inside or go outside (these
are the possibilities). Furthermore, you know if you usually
 Proceedings of the Eighth International Conference on Machine Learning and Cybernetics, Baoding, 12-15 July 2009
prefer to go outside or stay inside by good or bad weather,
and you know what you are aspiring to. Thus, as an
intelligent agent, you are able to make a decision, and
choose the optimal strategy, i.e. the strategy that
corresponds to your will. Thereby, as the objective is to
design a decision maker that can make as good decisions as
humans’, the HLANN will follow the same reasoning as a
man: it will receive two sorts of inputs, the conditions and
the possible choices, and it will be made up of two main
parts: one part that takes the decision, and another part that
expresses the feeling and the will of the intelligent agent.
2.2. Artificial Neural Network
As described above, the HLANN receives two kinds of
inputs: the conditions, as in [5], and the possible strategies
that the intelligent agent can choose.
The main part of this HLANN is the Decision Maker
network: an ANN - a multi or single layer perceptron, that
maps the inputs of the HLANN with its output: the chosen
strategy, modulo the addition of a small random variable in
order to explore all the possibilities. Once the choice is
made, the output of this network will be combined with the
environment variables to give the payoff of the intelligent
agent (the payoff is the result of the chosen strategy,
combined with other players’ strategies in Game Theory). If
we stopped the design of our HLANN here, we would have
the same ANN as the one that Harrald and Fogel used in [6]
to solve the iterated prisoner dilemma: a multi-layer feed
forward perceptron with 6 condition-inputs that correspond
to the previous moves of the two intelligent agents.
But, as described in the previous part, the HLANN is
also made up of another part that expresses the Feeling and
the Will of the intelligent agent. This part maps the payoff
with a Feeling and with a Will signal. If a payoff is very
lucrative for the intelligent agent, then the Feeling signal
will be high, and inversely if the payoff is bad. To do these
mappings, we use two ANN, multi or single layer
perceptrons, which map each possible payoff with a
“feeling-signal” first, ranged between -1 (for the worst
payoff), and 1 for the best payoff, and then with a
Will-signal. This Will-signal is also ranged between -1 and
1, and corresponds to a critic of the strategies: it expresses
if a strategy is wanted by the intelligent agent (Will of 1) or
not (will of -1). This Will-signal can be exactly the same as
the Feeling-signal, but it is not necessarily the case: if the
payoff that produces the best immediate Feeling always
leads to a very bad Feeling situation, it may not be
preferred to a situation that produces a good average
long-term Feeling.
Thereby, given a situation (conditions and possibilities)
1281
the objective of the HLANN is to find and choose the
strategy (output) that maximizes the Will-signal. Fig. 1
represents this whole Human-Like Neural Network.
Figure 1． Architecture of the Human-Like Artificial Neural
Network.
3. Learning Rules of the human-like ANN
The HLANN is constituted by two different neural
networks that have to be trained. The training algorithms of
these two parts are not the same. The Decision Maker
network is trained by reinforcement training, where the
reinforcement signal is the output of the Feeling-Will
network. If the intelligent agent has just two possible
strategies (i.e. the Decision Maker network has the choice
between two outputs–strategies), the learning rule of the
Decision Maker network could be one of the Selective
Bootstrap [7] or the Associative Reward-Penalty Neurons
[8] rules. Though, in most of strategic situations, the
intelligent agent has to choose between more than two
strategies. Furthermore, as we already said, the mapping
between the situations and the strategy to choose is often
nonlinear, so the Decision Maker network has to be a
multilayer network. In these cases, we use a reinforcement
learning using Back-Propagation, as described below, in
which the reinforcement signal produced by the
Feeling-Will network is back-propagated to correct the
weights of the Decision Maker network. Then, to complete
the training of the HLANN, the Feeling-Will network is
also updated, by using the back-propagation algorithm, in
order to make evolve the Will of the intelligent agent.
3.1. Reinforcement Learning using Back-Propagation
In this part, we introduce an adaptation of the
Back-propagation algorithm. In order to train the HLANN,
we want to minimize the function
F(r) = (1− r) 2 (1)
where r is the output of the Feeling-Will network, so the
 Proceedings of the Eighth International Conference on Machine Learning and Cybernetics, Baoding, 12-15 July 2009
reinforcement signal, as represented in Fig. 1. We recall that
r belongs to [-1,1] and is wanted to be equal to 1 (value for
which the chosen strategy is the most wanted strategy). The
training of the network will update the weights of each
layer k of the network (wkij) in order to minimize F(r). Thus,
by applying the steepest descent algorithm, one finds:
∂F ∂r
∆w k
ij = −α k = 2α (1− r) k
∂w ij ∂w ij
where α is the learning rate. Now, let us propagate this
expression through the HLANN until we reach the
output-layer of the Decision Maker network:
∂r ∂r ∂g ∂a M ∂n M
= (3)
∂w k ij ∂g ∂a M ∂n M ∂w k ij
where r, g, and a are defined in Fig. 2, M is the number of
layers of the Decision Maker, and nM is the net input before
its last layer’s transfer function fM. As we will need it below,
we already define this function fM as the logistic function.
Now, let us calculate every member of the right part of (3):
∂n M
= a M −1
∂w k ij
where aM-1 is the output of the M-1th layer of the Decision
Maker network. Then, we have:
∂a M ∂f M
= = a M (1− a M )
∂n M ∂n M
n th
Where a is the output of the n layer. For the other
members of (3), we will use a similar method as the one
used in the Stochastic Real-Valued Neuron (SRV) of
Gullapalli [9]: we cannot calculate the derivative of the
Figure 2． Architecture of the HLANN learning Algorithm.
payoff because the intelligent agent does not know this
function, so we express it as follows:
∂g g(achosen + v environment + v noise ) − g(achosen ) (6)
(a )≈
∂a chosen noise
where the vector venvironment corresponds to the direction of
(2) the evolution of the other variables of which the payoff
depends. The vector vnoise is a random vector that represents
a variation of the choice of the intelligent agent. Its norm is
|noise|, which is a random number selected according to a
mean-zero Gaussian distribution with very small standard
deviation (but noise≠0). So g(a) is the real payoff, given by
the combination between the chosen strategy a and the
Environment (other players’ strategies), while
g(a+venvironment + vnoise) is the expected payoff of a strategy
that would correspond to the output a+vnoise. As the
intelligent agent’s strategy is a and not a+vnoise, the value of
g(a+venvironment + vnoise) is not provided by the environment
itself but by an other neural network that approximates the
Environment, and that is trained by a supervised-learning
process, as the back-propagation algorithm, during an
exploration phase. The training set of this Environment
Approximation network is constituted by the outputs of the
Decision Maker network during the exploration phase and
(4)
the targets are given by the corresponding responses of the
environment.
Then, the last member of (3) is expressed by the same
method:
∂r r(g + noise) − r(g) (7)
≈
∂g noise
(5)
where r is the Will signal, and only depends on the payoff.
The noise is also selected according to a mean-zero
Gaussian distribution with very small standard deviation
(but noise≠0).
Thus, by replacing these expressions in (2), we obtain:
1282
 Proceedings of the Eighth International Conference on Machine Learning and Cybernetics, Baoding, 12-15 July 2009
∆w M = −αS M a M −1
where SM is the sensitivity of the Mth layer, and is given by:
S M = −2(1− r(g))
r(g + noise) − r(g) g(a + v env + v noise' ) − g(a) M
a (1− a M )
noise noise'
Then, as in the Back-propagation algorithm, the training of
the Decision Maker network is completed by
back-propagating this sensitivity through its layers.
3.2. Evolution of the Feeling-Will network
The Feeling-Will network represents the objectives of
the intelligent agent: its output is a Will-signal r(payoff)
that is maximal (equal to 1) for the most desired payoff and
minimal for the least desired payoff. In the Prisoner’s
dilemma, the goal of every player is obvious: earn as much
money as possible. Thus, the mapping between payoff and
feeling will be as follows: the payoff ‘5$’ will be awarded
by a feeling maximum (equal to 1), while the payoff ’0$’
will receive a feeling of -1. The willing signal will initially
just copy the feeling signal, in order to allow the intelligent
agent to get the best Feeling (feeling of 1). For example, for
the Prisoner dilemma, the training set given by the
environment would be:
P = [0 1 3 5]
T = [−1 −0.5 0.5 1]
where 0, 1, 3 and 5 are the possible payoffs, and T gives
their associated feeling.
However, in the Iterated Prisoner Dilemma, the goal of
every player is not to earn as much money as possible, but
to earn as much money as possible in the long-term, which
is different. To follow this objective, the Will-network has
to evolve, in order to allow the intelligent agent to choose
the strategy that maximizes its long-term feeling. As said
before, short-term objectives and long-term objectives can
appeal to different strategies, because some payoffs give
large immediate feelings, but then always lead to bad
situations, so the Will-signal has to take this into
consideration. To do so, the Will-Network has to play the
role of an adaptive critic unit, as introduced in [10]. It
produces a reinforcement signal that represents a weighted
sum of the actual feeling and the Expected future feelings
of each situation (each payoff). To produce this adaptive
critic signal the Will-Network is trained after each play
(move) by a supervised training algorithm (as
back-propagation) where the training set is constituted by
the payoffs, and their relative updated targets: the weighted
(8) sum of their immediate feeling and the Expected future
feeling from this situation (from this payoff). The
immediate feeling is directly provided by the Feeling
network, but the future reinforcement is given by the
following discounted sum:
(9) ∞
E Payoff (t) = E {∑ γ k feeling (t + k)}
k= 0
(11)
Of course, the intelligent agent cannot predict the values of
future Feelings (feeling(t+k)), but it can approximate it by
summing the weighted precedent feelings obtained from a
similar situation in the past. Then, the output target of the
Will-network for each situation (payoff) is updated after
each moves by the formula:
TPayoff = α . feeling Payoff (t) + (1 − α ).E Payoff
(12)
where α is a coefficient that indicates if the intelligent agent
prefers large immediate feeling (α=1) or good long-term
feeling (α=0). This Target vector is updated after each
plays in order to detect if a situation indeed leads to a good
or a bad long-term feeling. Retain that the Feeling network
is uniquely used to train the Will-network. Then, it allows
the HLAN to update its Will-network in order to produce a
(10) Will-signal (a reinforcement signal) that will lead it to the
strategy that optimizes its long-term or immediate feeling.
This tendency to long-term decision is undoubtedly an
evidence of intelligent behavior.
4. Experiments and Results
In the previous sections, we introduced a new HLANN
and its training algorithm. In this part, we implement this
ANN to a well-known problem of Game Theory: the
Iterated Prisoner Dilemma, introduced in the first part of
this paper. This dilemma describes well the problematic of
human decisions in economics and will allow us to see how
the HLANN can actually make decisions like humans. A
mathematical description of this dilemma was first
introduced in [4] and is displayed in Table 1.
Table 1. The payoff function of the Prisoner Dilemma: amount
of money that every player will receive, according to the
strategies of each player.
Player B
C D
C (3,3) (0,5)
Player A
(5,0) (1,1)
D
1283
 Proceedings of the Eighth International Conference on Machine Learning and Cybernetics, Baoding, 12-15 July 2009
To experiment the HLANN, we make it participate to
an IPD tournament against different well-known fixed
strategies that have been studied and used in [3] and [4]. In
this paper, we present the results of the HLANN against the
following strategies: Always Cooperate, Always Defect, Tit
for Tat (Cooperate on the first move, and then, for every
other moves, mimic the strategy of the other player at the
precedent move, which is the demonstrated Winning
strategy according to [3] and [4]), and Soft Majority
(cooperate as long as the opponent cooperated more than he
defected, else, defect).
For each game, The HLANN receives two inputs: the
two possible strategies, that are Cooperation,
mathematically represented by 1, and Defection,
represented by -1. Its output is, as in [6], a continuous
signal ranged between -1 (complete defection) and 1
(complete cooperation). Then, we give to the HLANN its
feeling and Will networks training sets (10), which are the
same for both at the beginning, because the goal is to
maximize the immediate feeling. Then, the HLANN begins
by an exploration phase, in order to discover the payoff
function, during which it maps the strategies of the two
opponents to the payoff of the HLANN, as shown in Fig. 3.
Figure 3. Approximation of the environment for the IPD.
Then, after this exploration phase, the game can begin:
the HLANN produces an output (a strategy). The
environment combines it with the strategy of the opponent
and gives to the HLANN its payoff. This payoff goes
through the Will-network to produce the reinforcement
signal. It also produces a feeling signal, which will then be
used to update the Will-network as explained in the
previous section. For example, Fig. 4 shows the Will-
network output after the game between two HLANN. We
can see that the most wanted situation (payoff) is no more
the largest payoff of 5 (as it was at the beginning), but the
payoff of 3. It is explained by the fact that the weighted
1284
sum of the immediate feeling and the Expectation of the
future feeling of the payoff of 3 has become larger than the
one of the payoff of 5. Thus, when the payoff is 5 for
example, the HLANN will change its weights in order to
increase the Will-signal, as expressed by (1), which
corresponds to a reduction of the payoff (it follows the
curve of Fig. 3 from the abscissa 5 to the abscissa 3).
According to Fig. 3, a reduction of the Payoff corresponds
to an increase in the output signal, i.e. a change of strategy
from strategy -1 (Defection) to strategy 1 (Cooperation). So
the HLANN changes its weights in order to increase its
output until the will-signal reaches the equilibrium, i.e. for
a payoff of 3. Then, the output will not change anymore. As
another example, if the opponent does not collaborate, then
the HLANN will never reach the payoffs of 3 or 5. Its
abscissa will then be between 0 and 3. Thus, according to
the derivative of the curve of Fig. 3 in this area, the
HLANN has to increase the payoff (in order to increase its
critic signal during the next move), which means, according
to Fig.3, a reduction of the output of the HLANN. So it will
change its weights in order to adopt the strategy -1
(Defection).
Figure 4. Will-network output after the game between two
HLANN.
This strategy is the one found by the HLANN itself,
without any expert knowledge. This strategy led the
HLANN to a large Win of the tournament, as shown in
Table 2 and Fig. 5. As one can see in Tab. 2, the HLANN is
the intelligent agent that, after having played against all the
other intelligent agents, had the maximum average score
(after 200 iterations in each game).
This win is not surprising, because the HLANN
doesn’t have a fixed strategy: it analyses by itself the
possibilities in each situation, chooses the one that
maximizes its feeling, and update its will-signal after every
move, in order to maximize the long-term feeling, as would
do a rational player.
 Proceedings of the Eighth International Conference on Machine Learning and Cybernetics, Baoding, 12-15 July 2009
Table 2. Averaged scores of most known intelligent agents.
Scores corresponds to the average amount of money the
intelligent agent would receive after playing 100 moves.
Intelligent Agent Average Score
(strategy)
HLANN 320
Tit for Tat 288
Soft Majority 281
Always Cooperate 248
Always Defect 237
What is very interesting is that the HLANN is able to
adapt itself to each situation, and to change its strategy,
which is a characteristic of intelligence.
Figure 5. Average of the score of the 5 strategies and average
of their opponents’ score.
Figure. 5 shows that the HLANN is the intelligent
agent that produces the best average score for him, and a
low score for his opponent. This is a typical ambitious
(human) strategy. Furthermore, as we just said, the HLANN
is able to adapt its strategy to the opponent. It is also able to
adapt its Will to the situation, and thus to find the only one
strategy that optimizes its long-term feeling, which is the
characteristic of an intelligent behavior.
5. Conclusions
In this paper, we have presented a human-inspired
approach that led to the design of an Artificial Neuron
Network that is capable of intelligent behaviors. This
human-like Artificial Neural Network requires a specific
reinforcement learning rule using Back-propagation that is
inspired from Gullapalli’s SRV. We have also presented a
simple example of the use of this human-like Artificial
Neural Network: the Iterated Prisoner Dilemma. This
example demonstrates the efficiency of the architecture and
1285
of the learning algorithms developed for this Human-Like
Artificial Neural Network. It also underlines the fact that
this human-like Artificial Neural Network is capable of
intelligent behaviors in decision making, without injecting
any expert-knowledge. It is indeed able to adapt itself to the
situation, and to find a strategy that optimizes its feeling.
Finally, because neural networks are universal function
approximators, this Human-Like Artificial Neural Network
can be a very useful tool in more complex situations, for
example continuous situations with more complex
environment, where one has to find the optimal strategy
without the intervention of an expert-knowledge.
Acknowledgements
The authors wish to acknowledge the support of
Elodie Durand and Pierre Nicol through many useful
discussions and Chen Xianmin, Zhang Yunfei and Zhu Jiayi
for their many advices.
References
[1] J. Von Neumann and O. Morgensten, Theory of
Games and Economic Behavior. Princeton, NJ:
Priceton Univ. Press, 1944.
[2] J. Maynard Smith, Evolution and the Theory of Games.
Cambridge University Press, 1982.
[3] R. Axelrod, The Evolution of Cooperation. New York:
Basic Books, 1984.
[4] R. Axelrod, More effective choice in the iterated
prisoner’s dilemma, J. Conflict Resolution, vol. 24, pp.
379-403, 1980.
[5] C. L. Tan, T. S. Quah, H. H. Teh, An Artificial Neural
Network that models Human Decision Making.
Computer, vol. 29, pp. 64-70, 1996.
[6] P.G Harrald and D. B. Fogel, Evolving continuous
behaviors in the iterated prisoner’s dilemma, Biosyst.,
vol. 37, n0. 1-2, pp. 135-145, 1996.
[7] B.Widrow, N. K. Gupta, and S. Maitra. Punish/reward:
Learning with a critic in adaptive threshold systems.
IEEE Transactions on Systems, Man, and Cybernetics,
vol. 5, pp. 455-465, 1973.
[8] A. G. Barto. Learning by statistical cooperation of
self-interested neuron-like computing elements,
Human Neurobiology, vol. 4, pp. 229-256, 1985.
[9] V. Gullapalli. A stochastic reinforcement algorithm
for learning real-valued functions. Neural Networks,
vol. 3, pp. 671-692, 1990.
[10] R. S. Sutton and A. G. Barto, Reinforcement learning:
An introduction. MIT Press, Cambridge, MA.