Evolving Bot AI in Unreal

,A.M. Mora, R. Montoya, J.J. Merelo, P. Garca-Sanchez1, P.A. Castillo1

J.L.J. Laredo, A. Martnez, andA.I.Esparcia3

+ Cloud Computing with EA

Mati Bot
Presentation Outline
First Person Shooters (FPS)
History of FPS
UnrealScript in a nutshell
2 evolutionary approaches to evolve Unreal bots
Genetic Algorithm
Genetic Programming
The experiment
Results
Future Work
Bonus: Cloud computing and EA
 The Problem-First Person Shooter
First Person View.
Inputs:
Can see the hands (which weapon is currently held).
Can see the enemies which are in front of the camera view.
Outputs:
Direction of movement.
Jump
Shoot/Hold fire
Change weapon
Goal:
Kill everyone.
Get 1 frag for each kill.
Don’t die.
Collect good weapons.
Collect good items.
History of FPS
Wolfenstein (ID software, 1987) – First single player FPS
Doom (ID software, 1988) – First Multiplayer FPS
Quake (ID software, 1992) – First Autonomous bots
 QuakeC-a program language for creating Quake Bot AI
Showed simple AI(Fuzzy logic at best)
Not suitable for GA
Unreal Tournament (Epic Mega Games, 1999)
UnrealScript-High level programming language
influenced by Java. Suitable for GA. [This paper is here]
Unreal Tournament 2004 (Epic Mega Games, 2004)
Unreal tournament 3 (EMG, 2009)
UnrealScript in a nutshell
Class declaration + inheritance:
class MyClass extends MyParentClass;
Here are the built-in variable types supported in UnrealScript:
 byte: A single-byte value ranging from 0 to 255.
 int: A 32-bit integer value.
 bool: A boolean value: either true or false.
 float: A 32-bit floating point number.
 string: A string of characters. (see Unreal Strings)
 constant: A variable that cannot be modified.
 enumeration: A variable that can take on one of several predefined named integer values. For example,
the ELightType enumeration defined in the Actor script describes a dynamic light and takes on a value
like LT_None, LT_Pulse, LT_Strobe, and so on.
 array<Type>: A variable length array of Type.
 struct: Similar to C structures, UnrealScript structs let you create new variable types that
contain sub-variables. For example, two commonly-used Unreal structs arevector, which consists
of an X, Y, and Z component; and rotator, which consists of a pitch, yaw, and roll component
UnrealScript in a nutshell
Supports repetition:
While,For,DoWhile
Conditionals:
If then else,case
Polymorphism
Built-in library (Math, String,Time,Vector,Debugging
and more…)
:Unreal Script Examples
 Declare an integer variable named "A".
 var int a;
 Declare a static array of 64 bytes named "Table".
 var byte Table[64];
 Declare a string variable named "PlayerName".
 var string PlayerName;
 Declare a float variable named "MaxTargetDist" and allow its value to
be modified from an UnrealEd property window.
 var() float MaxTargetDist;
:Unreal Script Examples

 Function to compute the factorial of a number.

 function int Factorial( int Number ) {
if( Number <= 0 )
;return 1
else
} ;return Number * Factorial( Number - 1 )
 Unreal Script-Bottom line
:The Good
1. Very similar to Java and C++ and very powerful.
2. Many games use the same language. (and this language is still being updated)
:The Bad
3. Array length must be less than 60. Arrays must be one dimensional. [1999]
4. Cannot run faster than real-time.
:The ugly
5. There isn’t a nice way to debug the code.

Link to the reference of UnrealScript if you want more info

 Genetic Bots
2 main approaches for the evolutionary process:
Evolve the parameters of the conditions that trigger the
transition from the current state to a new state
Evolve the Finite State Machine of the bot. (the edges
between the states and the states themselves.
Genetic Algorithm-based Bot
Firstly, 82 parameters were identified.
The difficulty:
 UnrealScript (as of 99) supports arrays of 60 floats.
 82 parameters are too much to evolve in the game.

Solution:
 Some parameters were not considered in the GA.
 Other parameters were represented as a function of another

parameter which is evolved in the GA.

In the end, 43 parameters (real numbers) were
evolved.
Genetic Algorithm-based Bot(cont.)
Representation: 43 floating points (normalized to [0,1]).
Crossover: 2-Point Crossover
Mutation: simple gene mutation (change the value of a
random gene (or none) by adding or subtracting a random
quantity in [0,1])
Selection: Ranked Based.
Scheme: Generational + Elitism
:Fitness Function
Genetic Programming-based Bot
Use GP to evolve new bot behavior.
First approach: Work with all possible input and
outputs. All states were considered.
Difficulty: UnrealScript can’t handle it due to its
constraints.
Solution:
Divide the search space of states into 2:
 Attacking, in which the bot decides between some possible

actions as: search, escape, move in a strategic way, attack and

how to do it (from distance, nearby, close, from a flank).
 Roaming, in which the bot searches for weapons and items
 Search Space-GP
UnrealScript functions:
Input functions (answer queries, ItemFound)
Output functions (state alteration,
Gotostate(‘Roaming’,’Camp’)
Considered only functions which are useful to
decision making. (HitWall, for instance, is used only
for animation).
Define rules on top of functions:
IF <INPUT-F1> OR <INPUT-F2> THEN <OUTPUT-F>
:Chromosome example
IF X1 THEN Y7, IF X3 AND X5 THEN Y2, IF X8 THEN
Y1, and IF X9 THEN Y9

Rule Set Parameters

 :Evolution parameters
Representation: Rule Tree as seen before + floating
points (normalized to [0,1])
Crossover: Node exchange(a valid one) + 2-Point
Crossover
Mutation: grows a random tree instead of a random
node + simple gene mutation (like before)
Selection: Ranked Based(like before)
Scheme: Generational + Elitism
Fitness Function
 The Experiment
8 players (bots-there was no technical difficulty for
letting humans to play) 1 of the 8 is the individual to
be tested and the rest of them came premade with the
system.
1 hour for a generation (the game is played In real-
time)
Each individual is playing until a threshold of defeats
is reached.
 The Experiment
The parameters for both methods:
 Results

The GA-Bots’ behavior was more coherent than the GPGA-Bots’.

The GA-Bots’ behavior was fixed and therefore performed better.
only the conditions for changing a state could be change.
GPGA-Bots didn’t perform as well because their behavior was
initially random and couldn’t beat the Unreal bots.
Both of the best bots that were evolved from both methods
eventually outperform all of the Unreal premade bots by
winning all matches.
 Future Work
Find the best Evolution parameters.
Implement this in a newer engine (like a new version
of Unreal)
Change the fitness function to be multi-objective
(Pareto front)
Cooperative bots (team of bots vs another team)
EA and Cloud Computing

Link to the cloud

 ?What is cloud computing
Hardware as Service

A Model for enabling convenient, on-demand network

:access to a shared pool of configurable computing resources
 Storage
 Servers

 Networks

 Applications

 Services
 Amazon Web Services
Cloud computing vendor.
Get access to as many computers as you like for a
cheap price.
Configure the image of each computer dynamically.
Pay only for what you need.
Provides storage as well. (and many other features)
Start a computer with service X

Start a computer with service Y

Access data
Start a computer with service Z

Communication channel
 AWS as a tool for EA Study
first approach

Fitness
Genetic evaluator
Algorithm

Fitness
evaluator

• Offers The ability to Fitness

run a GA experiment evaluator
faster than running on
a single PC.
 AWS as a tool for EA Study
trivial approach

Fitness
configuration evaluator

Genetic Fitness
Algorithm evaluator
• Offers The ability to
run a GA experiment
Fitness
and close the client
evaluator
application. Then you
can continue your work
on the same computer
and be more effective.
 AWS as a tool for EA Study
Advanced approach

Fi lua
ev
tn to
a
es r
s
Genetic
configuration Algorithm 1

ev itnes
al
F
ua tor
s
Experiment Genetic

e
Manager Algorithm 2

va
Fitn
lua s
tor
es
• Offers The ability to run Genetic

evaluator
Fitness
multiple GA experiments Algorithm 3
Shared
simultaneously (for Experiment
data
checking what fitness
function is better or a
new crossover operator
for instance)
 AWS as a tool for EA Study
More advanced approach

Fi lua
ev
tn to
a
Genetic

es r
s
GA algorithms
graph
Algorithm 1
configuration

ev itnes
al
F
ua tor
s
Experiment Genetic
Algorithm 2

eva ness
Manager

Fit
luat
or
• Uses the simplicity of the

evaluator
Fitness
cloud to connect different Genetic
GA nodes and share the Experiment Algorithm 3
data
genetic material
(Migration) with each
other by a configuration
made in advance.
The End