Markov Decision Process Tutorial

Intro to AI 096210

Erez Karpas
Faculty of Industrial Engineering & Managment
Technion

December 22, 2011

Erez Karpas

Markov Decision Process Tutorial

Markov Decision Process

A Markov Decision Process (MDP) is a stochastic planning

problem
Stationary Markovian Dynamics
The rewards and transitions only depend on current state

Fully observable
We might not know where were going, but we always know where
we are

Decision theoretic planning

We want to maximize expected reward

Erez Karpas

Markov Decision Process Tutorial

Markov Decision Process Formal Definition

A MDP consists of hS , A, R , T i
S is a finite set of states
A is a finite set of actions
R : S 7 [0, rmax ] is the reward function
Rewards are bounded

T : S A S 7 [0, 1] is the transition function

Probability of going from s to s0 after applying a is T (s, a, s0 )

Where is the initial state?

Erez Karpas

Markov Decision Process Tutorial

Markov Decision Process Example

Shamelessly stolen from Andrew Moore

You run a startup company. In every decision period, you must
choose between Saving money or Advertizing.
S = {Poor &Unknown, Poor &Famous, Rich&Unknown,
Rich&Famous}
A = {Save, Advertize}

(
R (s) =

if s = Poor & X

10

if s = Rich & X

T set next slide

Erez Karpas

Markov Decision Process Tutorial

Markov Decision Process Graphic Example

Erez Karpas

Markov Decision Process Tutorial

Markov Decision Process Solution

How do we solve an MDP?

What does a solution for an MDP look like?
A solution to an MDP is a policy : S 7 A
Given that Im in state s, I should apply action (s)
This is why we need full observability

What is an optimal policy?

Erez Karpas

Markov Decision Process Tutorial

Markov Decision Process Policy

We can compute the expected value of following fixed policy at

state s
V (s) = R (s) +

T (s, (s), s0 )V (s0 )

s0

is a discount factor
It makes sure the infinite sum converges
It can also be explained by interest rates, mortality, . . .

Value is immediate reward plus discounted expected future

reward

Erez Karpas

Markov Decision Process Tutorial

Markov Decision Process Optimal Policy Value

An optimal policy maximizes V (s) for all states

Is the optimal policy unique? No
Is the value of an optimal policy unique? Yes
We denote the value of an optimal policy at state s by V (s)
V (s) is unique

Erez Karpas

Markov Decision Process Tutorial

Markov Decision Process Using V

If we know V , we can simply choose the best action for each

state
The best action maximizes:
R (s) +

T (s, a, s0 )V (s0 )
s0

So we want to find V

Erez Karpas

Markov Decision Process Tutorial

Markov Decision Process Value Iteration

How do we find V ? Value Iteration

V 0 (s) = R (s)
V t (s) = R (s) + max
a

T (s, a, s0 )V t 1 (s0 )
s0

Converges: V t V
Stop when
max |V t (s) V t 1 (s)| <
s

Erez Karpas

Markov Decision Process Tutorial

Markov Decision Process Value Iteration Example

V 0 (s) =
V t (s) =

R (s)
R (s) + maxa s0 T (s, a, s0 )V t 1 (s0 )

= 0.9

t
0
1
2
3
4
5

V t (PU )
0
0
2.03
4.75
7.62
10.21

100

31.58

V t (PF )
0
4.5
8.55
12.2
15.07
17.46
...
38.6

V t (RU )
10
14.5
16.525
18.34
20.39
22.61

V t (RF )
10
19
25.08
28.72
31.18
33.2

44.02

54.2

Erez Karpas

Markov Decision Process Tutorial

Markov Decision Process Policy from Values

(s) :=

s
PU
PF
RU
RF

argmaxa R (s) + s0 T (s, a, s0 )V (s0 )

V
31.58
38.6
44.02
54.2

A
S
S
S

Erez Karpas

Markov Decision Process Tutorial

Markov Decision Process Policy Iteration

Another algorithm to find an optimal policy:

Initialize a policy arbitrarily

Evaluate V (s) for all states s S

3
4

0 (s) := argmaxa s0 T (s, a, s0 )V (s0 )

If 6= 0
:= 0
Goto 2

Erez Karpas

Markov Decision Process Tutorial

Markov Decision Process Policy Iteration Example

t
0
1
2

t (PU )

t (PF )

t (RU )

t (RF )

A
A
A

A
S
S

A
S
S

A
S
S

Erez Karpas

V t (PU )
0
31.58

V t (PF )
V t (RU )
0
10
38.6
44.02
Done

Markov Decision Process Tutorial

V t (RF )
10
54.2

Value Iteration vs. Policy Iteration

Which is better? It depends

VI takes more iterations than PI, but PI requires more time on
each iteration
Lots of actions? PI
Already got a fair policy? PI
Few actions, acyclic? VI
Also possible to mix

Erez Karpas

Markov Decision Process Tutorial

Solving MDP without the Model

What if we do not have access to the model?

We dont know transition probabilities T
We dont know reward function R

Then we cant compute a policy offline

We must choose an action online

Erez Karpas

Markov Decision Process Tutorial

Reinforcement Learning

The model
At every time step, the agent sees the current state s and the
applicable actions at s
After choosing an action to execute, the agent receives a reward

There are many RL algorithms

We will focus on Q-Learning

Erez Karpas

Markov Decision Process Tutorial

Q-Learning

We define Q : S A 7 [0, rmax ]

Q (s, a) is the best value we can expect after taking action a in
state s
Q (s, a) = R (s) +

Q (s0 , a0 )
T (s, a, s0 ) max
a
s0

Q (s, a) is immediate reward plus discounted expected future

reward if we choose the best action in the next state

Erez Karpas

Markov Decision Process Tutorial

Learning Q

Suppose our agent performed action a in state s

It moved to some state s0 , and got some reward R (s)
We can update Q (s, a):
Q (s, a) := Q (s, a) + R (s) + max Q (s0 , a0 ) Q (s, a)

a0

is the learning rate how much weight to give new vs. past
knowledge
Under some (realistic?) assumptions, Q-learning will converge to
optimal Q

Erez Karpas

Markov Decision Process Tutorial

Q-Learing: Exploration/Exploitation

Suppose were in the middle of Q-Learning

Were at state s
We have some estimate for Q (s, a), for any applicable action a
Which action to choose?
We can choose an action greedily the one which maximizes
Q (s , a )
But we might now know about the best action, and miss out
We want a policy that is greedy in the limit of infinite exploration
(GLIE)

Erez Karpas

Markov Decision Process Tutorial

GLIE Policies

Need to make exploitation more likely as more knowledge is

gained
One of the most popular GLIE policy is Boltzmann Exploration
Choose action a with probablity proportional to
eQ (s,a)/T
T is the temperature, which decreases with time

Erez Karpas

Markov Decision Process Tutorial