Reward-Based Deception with Cognitive Bias
Bo Wu, Murat Cubuktepe, Suda Bharadwaj, and Ufuk Topcu
Abstract— Deception plays a key role in adversarial or
strategic interactions for the purpose of self-defence and
survival. This paper introduces a general framework and
solution to address deception. Most existing approaches for
deception consider obfuscating crucial information to rational
adversaries with abundant memory and computation resources.
In this paper, we consider deceiving adversaries with bounded
arXiv:1904.11454v1 [cs.AI] 25 Apr 2019
rationality and in terms of expected rewards. This problem
is commonly encountered in many applications especially in-
volving human adversaries. Leveraging the cognitive bias of
humans in reward evaluation under stochastic outcomes, we
introduce a framework to optimally assign resources of a limited
quantity to optimally defend against human adversaries. Mod-
eling such cognitive biases follows the so-called prospect theory
from behavioral psychology literature. Then we formulate the
resource allocation problem as a signomial program to minimize
the defender’s cost in an environment modeled as a Markov
decision process. We use police patrol hour assignment as an
illustrative example and provide detailed simulation results
based on real-world data.
I. I NTRODUCTION
Deception refers to a deliberate attempt to mislead or
confuse adversaries so that they may take strategies that
are in the defender’s favor [1]. Deception can limit the
effectiveness of an adversary’s attack, waste adversary’s
resources and prevent the leakage of critical information [2].
It is a widely observed behavior in nature for self-defence
and survival. Deception also plays a key role in many aspects
of human society, such as economics [3], warfare [4], game
[5], cyber security [2] and so on.
In this paper, we focus on the scenario in which the adver-
sary acts in an environment where this interaction is modeled
as a Markov decision process (MDP) [6]. The adversary’s
aim is to collect rewards at each state of the MDP and the
defender tries to minimize the accumulated reward through
deception. Many existing approaches for deception rely on a
rational adversary with sufficient memory and computation
power to find its optimal policy [7], [1]. However, deceiving
an adversary with only bounded rationality [8], i.e., one
whose decisions may follow certain rules that deviate from
the optimal action [9], has not been adequately studied so
far. Deceiving an adversary with bounded rationality finds,
for example, its application in intrusion detection and pro-
tection [10] or public safety [11]. Different from obfuscating
sensitive system information to the adversary [12], [13], by
deception, we mean that the defender optimally assigns a
Bo Wu, Murat Cubuktepe, Suda Bharadwaj and Ufuk Topcu are with
the Department of Aerospace Engineering and Engineering Mechanics, and
the Oden Institute for Computational Engineering and Sciences, University
of Texas, Austin, 201 E 24th St, Austin, TX 78712. email: {bwu3,
mcubuktepe, suda.b, utopcu}@utexas.edu
limited resource to each state, such that the expected cost
from defender’s perspective (or equivalently, the reward for
the adversary) incurred by an adversary can be minimized,
even though the adversary is expecting more based on his
cognitively biased view of rewards.
To deceive a human more effectively, it is essential to
understand the human’s cognitive characteristics and what
affects his decisions (particularly with stochastic outcomes).
Works in behavior psychology, e.g. [14], suggested that
humans’ decision-making follows intuition and bounded
rationality. Empirical evidence has shown that humans
tend to evaluate gains and losses differently in decision-
making [15]. Humans tend to over-estimate the likelihood
of low-probability events and underestimate the likelihood
of high-probability events in a nonlinear fashion [16], [15].
Risk-sensitive measures, such as those in the so-called
prospect theory [16], capture such biases and are widely
used in psychology and economics to characterize human
preferences. Furthermore, humans tend to make decisions
that are often sub-optimal [17]. It is generally believed that
such sub-optimality is the result of intuitive decisions or
preferences that happen automatically and quickly without
much reflection [17], [14]. Human decisions are subject to
stochasticity due to the limited computational capacity and
inherent noise [18]. Consequently, human decisions are often
cognitively biased (have a different reward mechanism),
probabilistic (have a stochastic action selection policy) and
memoryless (only depends on the current state). These are
the very characteristics of human decision-making we expect
to account for in reward-based deception.
This paper investigates how one can deceive a human ad-
versary by optimally allocating limited resources to minimize
his rewards. We model the environment as an MDP to capture
the choices available to a human decision-maker and their
probabilistic outcomes. We consider opportunistic human
adversaries, i.e., they usually do not have significant planning
and only act based on immediately available rewards [11].
We describe the human adversary’s policy to select different
actions following the prospect theory and bounded rationality
[8]. We model both the adversary’s perceived reward and
defender’s cost (equivalently, the adversary’s reward from
the defender’s point of view) as functions of the resources
available at each state of the MDP. Additionally, we define
a subset of the states in the MDP as sensitive states that the
human adversary should be kept from visiting.
We then formulate the optimal resource allocation problem
as a signomial program (SP) to minimize the defender’s
cost. SPs are a special form of nonlinear programming
problems, and they are generally nonconvex. Solving non-
convex NLPs is NP-hard [19] in general, and a globally
optimal solution of an SP cannot be computed efficiently.
SPs generalize geometric programs (GP), which can be
transformed into convex optimization problems and then can
be solved efficiently [20]. In this paper, we approximate the
proposed SP to a GP. In numerical experiments, we show
that this approach obtain locally optimal solutions of the SP
efficiently by solving a number of GPs. We demonstrate the
approach with a problem on the assignment of police patrol
hour against opportunistic criminals [11].
The problem we study is closely related to the Stackelberg
security game (SSG) which consists of an attacker and a
defender that interact with each other. In SSG, the defender
acts first with limited resources and then the attackers play in
response [21]. SSG is a popular formalism to study security
problems against human adversaries. Early efforts focused
on one-shot games where an adversary can only take one
move [22] without considering human’s bounded rationality.
Then repeated SSG was considered in wildlife security [10]
and fisheries [23] where the defender and the adversary can
have repeated interaction. However, neither of these papers
considered how a human perceives probabilities, where the
existence of nonlinear probability weighting curves is a well-
known result in prospect theory [16]. Such phenomenon was
taken into account in [24] and [25]. But [24] only studied
one-shot games and [25] did not consider the adversaries
may move from place to place.
The rest of this paper is organized as the following.
We first provide the necessary preliminaries for stochastic
environment modeling, human cognitive biases and decision-
making in Section II. Then we formulate the human decep-
tion problem in terms of resource allocation in Section III
and show that it can be transformed into a signomial program
in Section IV. We propose the computational approach to
solve the signomial program in Section V. Section VI shows
simulations results and discusses their implications. We con-
clude our paper and discusses possible future directions in
Section VII.
II. P RELIMINARIES
A. Monomials, Posynomials, and Signomials.
Let V = {x1 , . . . , xn } be a finite set of strictly positive
real-valued variables. A monomial over V is an expression
of the form
f = c · xa1 1 · · · xann ,
where c ∈ R+ is a positive coefficient, and ai ∈ R are
exponents for 1 ≤ i ≤ n. A posynomial over V is a sum of
one or more monomials:
K
X power, therefore we only consider the memoryless policies.
g= ck · x1a1k · · · xannk . (1)
k=1
If ck is allowed to be a negative real number for any 1 ≤
k ≤ K, then the expression (1) is a signomial.
This definition of monomials differs from the standard al-
gebraic definition where exponents are positive integers with
no restriction on the coefficient sign. A sum of monomials
is then called a polynomial.
B. Nonlinear programs.
A general nonlinear program (NLP) over a set of real-
valued variables V is
minimize f (2)
subject to
gi ≤ 1, i = 1, . . . , m, (3)
hj = 1, j = 1, . . . , m, (4)
where f , gi , and hj are arbitrary functions over V , and m
and p are the number of inequality and equality constraints
of the program respectively.
C. Signomial programs and geometric programs.
A special class of NLPs known as signomial programs
(SP) is of the form (2)–(4) where f , gi and hj are signomials
over V , see Def. II-A. A geometric program (GP) is an SP of
the form (2)–(4) where f, gi are posynomial functions and hj
are monomial functions over V . GPs can be transformed into
convex programs [20, §2.5] and then can be solved efficiently
using interior-point methods [26]. SPs are non-convex pro-
grams in general, and therefore there is no efficient algorithm
to compute global optimal solutions for SPs . However, we
can efficiently obtain local optimal solutions for SPs in our
setting, as shown in the following sections.
In this paper, the adversary with bounded rationality moves
in an environment modeled as a Markov decision process
(MDP) [6].
D. Markov Decision Processes.
A (MDP) is a tuple M = (S, ν, A, T, U ) where
• S is a finite set of states;
• ν : S → [0, 1] is the initial state distribution;
• A is a finite set of actions;
0 0
• T (s, a, s ) := P (s |s, a). That is, the probability of
transiting from s to s0 with action a; and
+
• U (s) ∈ R is the utility function that assigns resources
with a quantity U (s) to state s.
At each state s, an adversary has a set of actions available
to choose. Then the nondeterminism of the action selection
has to be resolved by a policy π executed by the adversary.
A (memoryless) policy π : S × A → [0, 1] of an MDP M
is a function that maps every state action pair (s, a) where
s ∈ S and a ∈ A with probability π(s, a).
By definition, the policy π specifies the probability for the
next action a to be taken at the current state s. A bounded ra-
tional adversary is often limited in memory and computation
In an MDP, a finite state-action path is ω = s0 a0 s1 a1 ...,
where si ∈ S, ai ∈ A and T (si , ai , si+1 ) > 0. Given a
policy π, it is possible to calculate the probability of such
path Pπ (ω) as
Y
Pπ (ω) = ν(s0 ) π(si , ai )T (si , ai , si+1 ). (5)
i
 III. R EWARD -BASED D ECEPTION
We assume that an adversary with bounded rationality
moves around in an environment modeled as an MDP M =
(S, ν, A, T, U ). When the adversary is at a state s ∈ S, from
the defender’s point of view, the immediate reward for the
human adversary (or equivalently, the cost for the defender)
is
R(s) = g(U (s)) ∈ R+ ,
which is a function of allocated resource U (s). However, due
to the bounded rationality and cognitive biases, the perceived
immediate reward Rh (s) at state s by the adversary is a
different function of U (s), and is given by
Rh (s) = f (U (s)) ∈ R+ ,
where f is another function over U . For a given policy π,
expected rewards Qt (s) at each state s and time t with a
finite time horizon H can be evaluated as
XX
Qt (s) = R(s) + π(s, a)T (s, a, s0 )Qt+1 (s0 ), (6)
a s0
where t = 0..., H − 1, QH (s) = R(s). Therefore, Qt
represents the expect accumulated cost of the defender,
or equivalently, expected rewards for the human adversary
obtained from the policy π.
The defender’s objective is to optimally assign the re-
sources to each state to minimize his cost (equivalently, the
adversary’s reward) Q, where
X utility V (X) (i.e. the reward anticipated by a human) is the
Q= ν(s)Q0 (s), (7)
by designing the utility function U , where the resources are
P x∈O
of limited quantity, i.e., s U (s) = D. Also imagine that
there are set of sensitive states Ss ⊂ S that the adversaries
should be kept away from. Denote the set of paths that
reach Ss in H steps as Ω such that for each ω ∈ Ω where
ω = s0 a0 , ..., sN , we require N ≤ H, si ∈ / Ss , i < N and
sN ∈ Ss . In particular, given a policy π, P (♦≤H Ss ) can be
calculated as
X a at state s is
P (♦≤H Ss ) = P (ω). (8)
ω∈Ω
Problem 1: Given an MDP M = (S, ν, A, T, U ), time
horizon H, human reward function defined in (12) and the
policy π,
P design reward function U with a limited total
budget s U (s) = D, such that Q as defined in (7) can
be minimized ŝ and P (♦≤H Ss ) ≤ λ, which requires that
the probability to reach Ss in H steps should be no larger
than λ ∈ [0, 1], i.e.,
P (♦≤H Ss ) ≤ λ. (9)
Remark 1: Problem 1 studies how to optimally assign
the reward to trick the adversary into thinking that his
policy could obtain more rewards but in fact, the actual
expected reward is minimized with a low probability of
visiting sensitive states Ss .
A. Human Adversaries with Cognitive Biases
To solve Problem 1, it is essential to find the adversary’s
policy π. In this paper, we take human as the adversary with
bounded rationality who is opportunistic, meaning that he
does not have a specific attack goal nor plans strategically,
but is flexible about his movement plan and seek opportu-
nities for attacks [27]. Those attacks may incur rewards to
the human adversary and consequently certain costs for the
defender. The process of human decision-making typically
follows several steps [28]. First, a human recognizes his
current situation or state. Second, he will evaluate each
available action based on the potential immediate reward it
can bring. Third, he will select an action following some
rules. Then he will receive a reward and observe a new state.
In this section, we will introduce the modeling framework
for the second and third step.
For a human with bounded rationality, the value of a
reward from an action is a function of the possible outcomes
and their associated probabilities. The prospect theory de-
veloped by Kahneman and Tversky [16] is a frequently used
modeling framework to characterize the reward perceived by
a human. Prospect theory claims that humans tend to over-
estimate the low probabilities and underestimate the high
probabilities in a nonlinear fashion. For example, between
1
winning 100 dollar with 100 probability and nothing else, or
1 dollar with probability 1, humans tend to prefer the former,
even though both have the same expectation.
Given X as the discrete random variable that has a
finite set of outcomes O, a general form of prospect theory
following. X
V (X) = v(x)w(p(x)), (10)
where v(x) ∈ R denotes the reward perceived by a human
from the outcome x. The probability p(x) to get the outcome
x is weighted by a nonlinear function w that captures
the human tendency to over-estimate low probabilities and
under-estimate high probabilities.
The expected immediate reward ra (s) to perform an action
X
ra (s) = R(s0 )T (s, a, s0 ). (11)
s0
However, according to prospect theory, from a human’s
perspective, the perceived expected immediate reward rah (s)
is different. Let Xs,a be the random variable for the outcome
Os,a of executing action a at state s. We have Os,a =
{x0s |T (s, a, s0 ) > 0} where xs0 denotes the event that the
state transits from s to s0 with an action a. The distribution
of Xs,a is defined as follows.
p(xs0 ) = T (s, a, s0 ), ∀xs0 ∈ Os,a .
The human perceived reward v(xs0 ) for the outcome xs0
depends on U (s0 ) received from reaching the state s0 , which
is denoted by
v(xs0 ) = Rh (s0 ) = f (U (s0 )).
 U (s2 ) = 2 U (s3 ) = 2.5
s2 s3 X
minimize Q = ν(s)Q0 (s) (15)
b, 0.5
a, 0.9 subject to
b, 0.5
U (s1 ) = 5 s1 s0 s4 U (s4 ) = 2.5 ∀s ∈ S, t ∈ {0, . . . , H − 1},
a, 0.1 XX
Qt (s) ≥ R(s) + π(s, a)T (s, a, s0 )Qt+1 (s0 )
a∈A s0 ∈S
Fig. 1. A simple example for sub-optimality with human cognitive biases (16)
∀s ∈ S, t ∈ {0, . . . , H − 1},
XX
As a result, rah (s) is denoted by Pt (s) ≥ π(s, a)T (s, a, s0 )Pt+1 (s0 ) (17)
X a∈A s0 ∈S
rah (s) = v(xs0 )w(p(xs0 )) ∀t ∈ {0, . . . , H}, ∀s ∈ Ss , Pt (s) = 1 (18)
xs0 ∈Os,a X
X (12) ∀t ∈ {0, . . . , H}, ν(s)Pt (s) ≤ λ (19)
= f (U (s0 ))w(T (s, a, s0 )). s∈S
s0
∀s ∈ S, a ∈ A,
An empirical form of w is the following [16]. X X
π(s, a) f (U (s0 ))w(T (s, a0 , s0 ))
pγ a0 ∈A s0 ∈S
w(p) = 1 , γ > 0. (13) X
(pγ + (1 − p)γ ) γ
= f (U (s0 ))w(T (s, a, s0 )) (20)
s0 ∈S
Given an MDP as depicted in Figure 1, where S =
{s0 , . . . , s4 }, A = {a, b}. We assume that R(s) = U (s), ∀s ∈ S, R(s) = g(U (s)) (21)
Rh (s) = U (s)0.88 , γ = 0.6 in (13). It can be found from (11)
X
U (s) = D, (22)
and (12) that ra (s0 ) = 2.3, rah (s0 ) = 2.0678, rb (s0 ) = 2.5 s∈S
and rbh (s0 ) = 1.8617. Since rah (s0 ) > rbh (s0 ), suppose a where variables R(s) are for rewards in each state s, U (s)
human is at s0 , from human’s perspective, he will prefer are for utilities in each state s, π(s, a) are for the probability
the action a. However, ra (s0 ) < rb (s0 ) which indicates that of taking action a in state s are for each state and action,
action a actually has more expected immediate rewards. Qt (s) are for the expected reward of the state s and time
Remark 2: In this example, the rewards are already given, step t, and Pt (s) are for the probability of reaching the set
and it can be seen that the human could make a sub-optimal of target states Ss in each state s and time step t.
decision. It illustrates how cognitive bias can deviate the The objective in (15) minimizes the accumulated expected
human behavior from optimal. reward from the initial state distribution ν(s) over a time
After evaluating the outcome of each candidate action a horizon H. In (16), we compute Qt (s) by adding the
by rah (s), a human then needs to make an action selection. immediate reward in state s and the expected reward of the
Humans are known to only have quite limited cognitive successor states according to the policy variables π(s, a) for
capabilities. Human’s policy π to choose an action can be each action a. The probability of reaching each successor
described as a random process that biases toward the actions state s0 depends on the policy variables π(s, a) in each state
of high rah (s), such that s and action a. Similar to the constraint in (16), the variables
rah (s) Pt (s) are assigned to the probability of reaching the set of
π(s, a) = P h
, (14) target states Ss from state s and time step t in (17).
a0 ra0 (s)
The probability of reaching any state s ∈ Ss in each
where π(s, a) denotes the probability of executing the action horizon from the states in Ss is set to 1 as in (18). The
a at state s. Such a bounded rational behavior has been constraint in (19) assures that the probability of reaching
observed in humans, such as urban criminal activities [29]. any state s ∈ Ss from the initial state distribution ν(s) is
Intuitively, it implies that human selects the action a oppor- less than λ. The constraint in (20) computes the policy using
tunistically at each state s with the probability proportional the model in (14). We give the relationship between rewards
to the perceived immediate reward rah (s). and utilities in (21). Finally, (22) gives the total budget for
Now we are ready to redefine Problem 1 as follows. utilities.
Problem 2: Solve Problem 1 for π defined as (14). The constraint in (16) and (17) are convex constraints,
because the functions in the right hand sides are posynomial
IV. S IGNOMIAL P ROGRAMMING F ORMULATION functions, and the functions in the left hand sides are mono-
Given an MDP M, time horizon H, human reward func- mial functions. The constraints in (18) and (19) are affine
tion and policy as defined in (12) and (14), the solution of constraints, therefore they are convex. The constraints in (20)
the Problem 1 can be computed by solving the following and (22) are equality constraints with posynomials, therefore
signomial program. The g and f are assumed to be monomial they belong to the class of signomial constraints, and they
functions of U for our solution method. are not convex. In the literature, there are various methods
to deal with the nonconvex constraints to obtain a locally the reachability constraint in (19), we change the objective
optimal solution including sequential convex programming, in (15) as follows:
convex-concave programming, branch and bound or cutting
plane methods [30], [20], [31]. minimize Q + δ · τ (25)
where δ is a positive penalty parameter that determines
V. C OMPUTATIONAL A PPROACH FOR THE S IGNOMIAL the violation rate for the soft constraint in (23). In our
P ROGRAM formulation, we increase δ after each iteration to satisfy the
In this section, we discuss how to compute a locally reachability constraint.
optimal solution efficiently for Problem 1 by solving the We stop the iterations when the change in the value of Q
signomial program in (15)–(22). We propose a sequential is less than a small positive constant . Intuitively,  defines
convex programming method to compute a local optimum the required improvement on the objective value for each
of the signomial program in (15)–(22), following [20, §9.1], iteration; once there is not enough improvement, the process
solving a sequence of GPs. We obtain each GP by replacing terminates.
signomial constraints in equality constraints of the SGP sig-
VI. N UMERICAL E XPERIMENT
nomial program in (15)–(22) with monomial approximations
of the functions. Let us consider an urban security problem, where a
criminal plans his next move randomly based on his local
A. Monomial approximation information on the nearby locations that are protected by
Given a posynomial f , a set of variables {x1 , . . . , xn }, police patrols. Such a criminal is opportunistic, i.e, he is
and an initial point x̂, a monomial approximation [20] fˆ for not highly strategic by conducting careful surveillance and
f around x̂ is rational planning before making moves. It is known that this
!ai kind of opportunistic adversaries contribute to the majority
n
Y x i
of the urban crimes [32]. For prevention and protection, each
∀i.1 ≤ i ≤ n fˆ = f [x̂] , location should be assigned a certain police patrol hours. Due
i=1
x̂(xi )
to the limited amount of police resources, the total number
x̂(xi ) ∂f of patrol hours is limited as well.
where ai = [x̂].
f [x̂] ∂xi
Intuitively, a monomial approximation of a posynomial f
around an initial point x̂ corresponds to an affine approxima-
tion of the posynomial f . Such an approximation is provided
by the first order Taylor approximation of f , see [20, §9.1]
for more details.
For a given instantiation of the utility and policy variables
U (s) and π(s, a), we approximate the SP in (15)–(22) to
obtain a GP as follows. We first normalize the utility values
to ensure that they sum up to D. Then, using those utility
values, we compute the policy according to constraint in (20).
After the policy comptutation, we compute a monomial
approximation of each posynomial term in the constraints
(20) and (22) around the previous instantiation of the utility
and policy variables. After the approximation, we solve the
approximate GP. We repeat this procedure until the procedure
converges.
One key problem with this approach is, we require an
initial feasible point to the signomial problem in (15)–(22),
which may be hard to find because of the reachability
constraint in (19). Therefore, we introduce a new variable
τ and we replace the reachability constraint in (19) by the
following constraints:
X
∀t ∈ {0, . . . , H}, ν(s)Pt (s) ≤ λ · τ (23) Fig. 2. The 35 intersections in North East of San Francisco. The map
s∈S
is obtained from Google map. The shaded area is a circle with a 500
feet radius. The numbering of the states starts from the bottom left corner
τ ≥ 1. (24) and goes from left to right in every row. The number beside each location
indicates the number of crimes in that area.
By replacing the reachability constraint, we ensure that
any initial utility function and policy is feasible to the Figure 2 shows 35 intersections in San Francisco, CA with
signomial program in (15)–(24). To enforce the feasability of 7 rows and 5 columns. We use an MDP M = (S, ν, A, T, U )
to describe the network of the set of intersections S. The
number C(s) of crimes that occurred in the first four weeks
of October, 2018 within 500 feet of each interaction s
is shown in Figure 2. The crime data are obtained from
https://www.crimemapping.com/map/ca/sanfrancisco. The
criminal can choose to move left, right, up or down to the
immediate neighboring intersections. Consequently, there
are four actions available. The execution of each action
will lead the human to its intended neighborhood of the
intersection with a high probability (≥ 0.95) and small
probability to other neighboring intersections to account for
unexpected change of movement plan.
Initially, the criminal has equal probability to appear at any
1
state, i.e., ν(s) = 35 for any s ∈ S. The utility U (s) denotes
the number of police patrol hours that should be allocated to
the vicinity of each intersection.
P The total number of police
patrol hours is D = U (s) = 400. If a location s is
assigned with U (s) patrol hours, its reward to the criminal
(equivalently, the cost to the defender) is
C(s)
R(s) = .
U (s)
Intuitively, it means that the reward to the human adversary,
from the defender’s point of view, is proportional to the
crime rate indicated by C(s) and inversely proportional to the Fig. 3. The distribution of utility U .
police patrol hours. The reward from the human adversary’s
view is evaluated as
f (U (s)) = R(s)0.88 , U (s), assigned to each location s. In Figure 3, U (s) is shown
with a logarithmic scale for better illustration. As the color
which is a function commonly seen in the literature to bar at the bottom of the figure indicates, the closer the color
describe how human biases the reward [15]. at each location is to the right side of this bar, the higher
Initially, the criminal is at s with probability ν(s), where patrol hours are assigned. For example, the state at (3, 1)
he tries to plan his move over the next H steps. The objective (the third state from the first row), where C(s) = 48 gets
is to assign the police patrol hours to each state, such that assigned patrol hour equals 106 which is approximately 2 in
the expected accumulated reward in H steps received by the logarithmic scale. Therefore, its color is yellow in Figure 3
criminal is minimized. The sensitive states Ss = {3, 14, 33} as indicated by the right tip of the color bar. Together with
should be visited with a probability no larger than λ = 0.3, Figure 2, it can be observed that sensitive places and places
i.e. with a higher number of crimes get assigned more patrol
P (♦≤H Ss ) ≤ 0.3. hours. Consequently, the rewards at those states are fairly
The sensitive states are also shown as blue circles in Figure low to discourage the criminal from visiting it. The cost at
2. each location is proportional to the crime rate and inversely
We formulate the problem as a signomial program. From proportional to the police patrol hours. The patrol hours
an initial uniform utility distribution, we instantiate the poli- assigned to each place intends to minimize the expected cost
cies and reward functions. Then, from the initial values, we incurred by the human adversary.
linearize the signomial program in (15)–(25) to a geometric
VII. C ONCLUSION
program. We parse the geometric programs using the tool
GPkit [33], and solve them using the solver MOSEK [34]. This paper introduces a general framework for deceiving
We set  = 10−4 for convergence tolerance. All experiments adversaries with bounded rationality in terms of the obtained
were run on a 2.3 GHz machine with 16 GB RAM. The reward minimization. Leveraging the cognitive bias of the
procedure converged after 32 iterations for a problem with human from well-known prospect theory, we formulate the
horizon length T = 20 in 230.06 seconds. The expected reward-based deception as a resource allocation problem
reward Q from the initial state distribution is 117.15, and the in Markov decision process environment and solve as a
reachability probability of the sensitive states from the initial signomial program to minimize the adversary’s expected
state distribution is 0.192, which satisfies the reachability reward. We use police patrol hour assignment as the illustra-
specification. tive example and show the validity of our propose solution
The result is shown in Figure 3. Different colors at each approach. It opens doors for further research on the topic to
intersection show the number of patrol hours, i.e, the resource consider the scenarios where defender can move around and
react to the human adversaries in real time, and the human [24] R. Yang, C. Kiekintveld, F. OrdóñEz, M. Tambe, and R. John,
adversary has a learning capability to adapt the defender’s “Improving resource allocation strategies against human adversaries in
security games: An extended study,” Artificial Intelligence, vol. 195,
deceiving policy. pp. 440–469, 2013.
[25] D. Kar, F. Fang, F. Delle Fave, N. Sintov, and M. Tambe, “A game of
thrones: when human behavior models compete in repeated stackelberg
R EFERENCES security games,” in Proceedings of the 2015 International Conference
on Autonomous Agents and Multiagent Systems. International Foun-
[1] M. H. Almeshekah and E. H. Spafford, “Cyber security deception,” in dation for Autonomous Agents and Multiagent Systems, 2015, pp.
Cyber deception. Springer, 2016, pp. 23–50. 1381–1390.
[2] J. Pawlick, E. Colbert, and Q. Zhu, “A game-theoretic taxonomy and [26] S. Boyd and L. Vandenberghe, Convex optimization. Cambridge
survey of defensive deception for cybersecurity and privacy,” arXiv university press, 2004.
preprint arXiv:1712.05441, 2017. [27] Y. D. Abbasi, M. Short, A. Sinha, N. Sintov, C. Zhang, and M. Tambe,
[3] S. Bonetti, “Experimental economics and deception,” Journal of Eco- “Human adversaries in opportunistic crime security games: Evaluating
nomic Psychology, vol. 19, no. 3, pp. 377–395, 1998. competing bounded rationality models,” in Proceedings of the Third
[4] T. Holt, The deceivers: Allied military deception in the Second World Annual Conference on Advances in Cognitive Systems ACS, 2015, p. 2.
War. Simon and Schuster, 2010. [28] K. Doya, “Modulators of decision making,” Nature neuroscience,
[5] E. Morgulev, O. H. Azar, R. Lidor, E. Sabag, and M. Bar-Eli, vol. 11, no. 4, p. 410, 2008.
“Deception and decision making in professional basketball: Is it [29] M. B. Short, M. R. D’orsogna, V. B. Pasour, G. E. Tita, P. J.
beneficial to flop?” Journal of Economic Behavior & Organization, Brantingham, A. L. Bertozzi, and L. B. Chayes, “A statistical model
vol. 102, pp. 108–118, 2014. of criminal behavior,” Mathematical Models and Methods in Applied
Sciences, vol. 18, no. supp01, pp. 1249–1267, 2008.
[6] M. L. Puterman, Markov decision processes: discrete stochastic dy-
[30] R. E. Moore, “Global optimization to prescribed accuracy,” Computers
namic programming. John Wiley & Sons, 2014.
& mathematics with applications, vol. 21, no. 6-7, pp. 25–39, 1991.
[7] K. Horák, Q. Zhu, and B. Bošanskỳ, “Manipulating adversary’s belief: [31] E. L. Lawler and D. E. Wood, “Branch-and-bound methods: A survey,”
A dynamic game approach to deception by design for proactive Operations research, vol. 14, no. 4, pp. 699–719, 1966.
network security,” in International Conference on Decision and Game [32] P. J. Brantingham and G. Tita, “Offender mobility and crime pattern
Theory for Security. Springer, 2017, pp. 273–294. formation from first principles,” in Artificial crime analysis systems:
[8] H. A. Simon, “Models of man; social and rational.” 1957. using computer simulations and geographic information systems. IGI
[9] C. F. Camerer, Behavioral game theory: Experiments in strategic Global, 2008, pp. 193–208.
interaction. Princeton University Press, 2011. [33] E. Burnell and W. Hoburg, “Gpkit software for geometric pro-
[10] R. Yang, B. Ford, M. Tambe, and A. Lemieux, “Adaptive resource allo- gramming,” https://github.com/convexengineering/gpkit, 2018, version
cation for wildlife protection against illegal poachers,” in Proceedings 0.7.0.
of the 2014 international conference on Autonomous agents and multi- [34] M. ApS, The MOSEK optimization toolbox for PYTHON. Version 7.1
agent systems. International Foundation for Autonomous Agents and (Revision 60), 2015. [Online]. Available: http://docs.mosek.com/7.1/
Multiagent Systems, 2014, pp. 453–460. quickstart/Using MOSEK from Python.html
[11] C. Zhang, A. X. Jiang, M. B. Short, P. J. Brantingham, and
M. Tambe, “Defending against opportunistic criminals: New game-
theoretic frameworks and algorithms,” in International Conference on
Decision and Game Theory for Security. Springer, 2014, pp. 3–22.
[12] B. Wu and H. Lin, “Privacy verification and enforcement via belief
abstraction,” IEEE Control Systems Letters, vol. 2, no. 4, pp. 815–820,
Oct 2018.
[13] P. Masters and S. Sardina, “Deceptive path-planning,” in Proceedings
of the 26th International Joint Conference on Artificial Intelligence.
AAAI Press, 2017, pp. 4368–4375.
[14] D. Kahneman, Thinking, Fast and Slow. Macmillan, 2011.
[15] A. Tversky and D. Kahneman, “Advances in prospect theory: Cumu-
lative representation of uncertainty,” Journal of Risk and uncertainty,
vol. 5, no. 4, pp. 297–323, 1992.
[16] D. Kahneman and A. Tversky, “Prospect theory: An analysis of
decision under risk,” in Handbook of the fundamentals of financial
decision making: Part I. World Scientific, 2013, pp. 99–127.
[17] E. Norling, “Folk psychology for human modelling: Extending the bdi
paradigm,” in Proceedings of the Third International Joint Conference
on Autonomous Agents and Multiagent Systems-Volume 1. IEEE
Computer Society, 2004, pp. 202–209.
[18] P. B. Reverdy, V. Srivastava, and N. E. Leonard, “Modeling human
decision making in generalized gaussian multiarmed bandits,” Pro-
ceedings of the IEEE, vol. 102, no. 4, pp. 544–571, 2014.
[19] D. S. Hochbaum, “Complexity and algorithms for nonlinear optimiza-
tion problems,” Annals of Operations Research, vol. 153, no. 1, pp.
257–296, 2007.
[20] S. Boyd, S.-J. Kim, L. Vandenberghe, and A. Hassibi, “A tutorial on
geometric programming,” Optimization and Engineering, vol. 8, no. 1,
2007.
[21] B. An and M. Tambe, “Stackelberg security games (ssg) basics and
application overview,” Improving Homeland Security Decisions, p.
485, 2017.
[22] M. Jain, J. Tsai, J. Pita, C. Kiekintveld, S. Rathi, M. Tambe, and
F. Ordónez, “Software assistants for randomized patrol planning for
the lax airport police and the federal air marshal service,” Interfaces,
vol. 40, no. 4, pp. 267–290, 2010.
[23] W. B. Haskell, D. Kar, F. Fang, M. Tambe, S. Cheung, and E. Denicola,
“Robust protection of fisheries with compass.” 2014.