Statistical Models in Simulation

CP463 Simulation
Dr. Ilias S. Kotsireas, ikotsire@wlu.ca
Wilfrid Laurier University
Department of Computing
1. Introduction
2. Basic Probability: Terminology & Concepts
3. Useful Statistical Models
4. Discrete Distributions
5. Continuous Distributions
6. Poisson Processes
7. Empirical Distributions
1
Introduction
1. To model real-world phenomena we have to
represent actions of the entities within the
system. These actions are usually unpredictable.
2. It turns out that some statistical models are
suitable to describe such actions.
3. To model a random process (e.g.time it takes
to repair a machine) the modeler has to select
an appropriate probability distribution.
4. steps to select an appropriate p.d.
sample the process
select a known distribution (experience, spe-
cial software)
estimate the values of the parameters of
the distribution
test to see how good is the t (relate to
the concept of interpolation)
2
Basic Probability
Discrete Random Variable ==
random variable (X) with a nite (x
1
, . . . , x
n
) or
countably innite (
0
) set of possible values. This
set is called the range of X and is denoted by R
X
.
Example:
X = # orders each day at a factory.
R
X
= {0, 1, 2, . . . , }
Each possible outcome x
i
has an associated prob-
ability of appearance p(x
i
) = P(X = x
i
).
These probabilities p(x
i
) satisfy:
p(x
i
) 0 for all i.

i=1
p(x
i
) = 1.
3
Probability Distribution of X ==
collection of pairs (x
i
, p(x
i
)), i = 1, 2, . . ..
Probability Mass Function of X (pmf) ==
p(x
i
)
Example:
toss a die. X = # spots on the upper face after
toss.
R
X
= {1, 2, 3, 4, 5, 6}
die is unbiased
these 6 events are equiprobable
p(1) = p(2) = p(3) = p(4) = p(5) = p(6) =
1
6
.
4
assume die is biased: p(i) is proportional to i.
p(i) = k i, for i = 1, 2, 3, 4, 5, 6.
p(1) +p(2) +p(3) +p(4) +p(5) +p(6) = 1
k(1 +2 +3 +4 +5 +6) = 1 k =
1
21
discrete probability distribution for biased die toss:
x
i
1 2 3 4 5 6
p(x
i
)
1
21
2
21
3
21
4
21
5
21
6
21
5
Continuous Random Variable ==
random variable (X) with a range R
X
which is an
interval or a collection of intervals.
the probability that X lies in an interval [a, b] is
given by:
P(a X b) =
_
b
a
f(x)dx
Probability Density Function (pdf) == f(x)
The pdf f(x) satises:
f(x) 0 for all x R
X
.
_
R
X
f(x)dx = 1.
f(x) = 0, if x R
X
.
6
Remark:
for a specic x
0
we have
_
x
0
x
0
f(x)dx = 0 and
therefore P(X = x
0
) = 0.
in particular:
P(a X b) = P(a < X b) = P(a X < b) =
P(a < X < b)
Example:
life of a laser device is given by a continuous ran-
dom variable taking all values in the range x 0.
pdf in years:
f(x) =
_
_
_
1
2
e
x/2
, x 0
0, x < 0
the probability that the life of the device is be-
tween 2 and 3 years is:
P(2 X 3) =
_
3
2
f(x)dx =
1
2
_
3
2
e
x/2
dx = . . .
7
Cumulative Distribution Function (cdf) == F(x)
measures the probability that the random variable
X assumes a value less than or equal to x
F(x) =

x
i
x
p(x
i
)
F(x) =
_
x

f(t)dt
Properties of the cdf F(x): (justied intuitively
by positivity)
F is a nondecreasing function.
a < b F(a) F(b).
lim
x
F(x) = 1.
lim
x
F(x) = 0.
The cdf is useful because it can be used to answer
probability questions about X.
P(a < X b) = F(b) F(a), a < b
8
Example: cdf of the biased die tossing example:
x (, 1) [1, 2) [2, 3) [3, 4) [4, 5) [5, 6) [6, )
F(x) 0
1
21
3
21
6
21
10
21
15
21
21
21
Example: cdf of the life of laser device example:
F(x) =
1
2
_
x
0
e
t/2
dt = 1 e
x/2
probability that the life of the device will be less
than 2 years:
P(0 X 2) = F(2) F(0) = 1 e
1
probability that the life of the device is between
2 and 3 years:
P(2 X 3) = F(3)F(2) = (1e
3/2
)(1e
1
)
9
Expectation, Mean, First Moment denoted: E(X),
E(X) =

i
x
i
p(x
i
) X is discrete
E(X) =
_

xf(x)dx X is continuous
n-th Moment denoted: E(X
n
)
E(X
n
) =

i
x
n
i
p(x
i
) X is discrete
E(X
n
) =
_

x
n
f(x)dx X is continuous
Variance denoted:V (X),
2
(spread of X values
around )
V (X) = E((X E(X))
2
) = E(X
2
) E(X)
2
Standard Deviation denoted:
10
Example: biased die tossing.
E(X) = 1
_
1
21
_
+. . . +6
_
6
21
_
=
91
21
= 4.33
E(X
2
) = 1
2
_
1
21
_
+. . . +6
2
_
6
21
_
= 21
(quick justication, sum of cubes)
V (X) = E(X
2
)E(X)
2
= 21
_
91
21
_
2
= 2118.78 = 2.22
=
_
V (X) = 1.49
11
Example: life of laser.
E(X) =
1
2
_

0
xe
x/2
dx = 2 years
E(X
2
) =
1
2
_

0
x
2
e
x/2
dx = 8 years
V (X) = E(X
2
) E(X)
2
= 8 2
2
= 4 years
2
=
_
V (X) = 2 years
(reminder: Integration by parts)
12
Useful Statistical Models
In many cases the modeler needs to introduce
probabilistic events in the course of a simulation:
1. interarrival and service times in a queueing
system.
2. time between demands and lead time

in an
inventory system.
3. time to failure in a reliability model.
In each of these cases, the modeler must generate
random events according to a known statistical
model.
We illustrate the use of some statistical models
in particular categories of simulation examples.

time between placing and receiving an order

13
Queueing Systems Service times may be con-
stant or random. In the latter case we use:
the exponential distribution,
the truncated normal distribution (existence
of a lower bound for the random variable),
gamma distribution,
Weibull distribution (when service times are
too large to be described by an exponential
distribution)
Inventory Systems usually we model 3 ran-
dom variables: # units demanded per order,
# time between demands, # lead time.
in practice, one models lead-time by a gamma
distribution.
the demand time is modeled by the Poisson
distribution, and/or the negative binomial dis-
tribution.
14
Reliability and Maintainability the time-to-
failure distribution is modeled with an expo-
nential distribution.
gamma distribution models systems with many
components, where each component has an
exponential time to failure.
Weibull distribution models systems with many
components, where failure is due to the most
serious of a large number of defects
normal distribution models systems with many
components, where most failures are due to
wear.
lognormal distribution is used to describe fail-
ure of some types of components, increasingly
used in reliability models.
15
Limited (Incomplete) Data Simulation be-
gins before the data collection phase has been
completed.
Uniform distribution: a time is known to be
random but there is no other information avail-
able.
Triangular distribution: used when assump-
tions are made about the minimum, maximum
and modal values of the random variable.
Beta distribution: provides a variety of dis-
tributions on the unit interval, that can be
shifted to an arbitrary interval.
Other distributions
Bernoulli, binomial, hyperexponential ...
16
Discrete Distributions (4)
Discrete random variables describe random phe-
nomena in which only integer values can occur.
Bernoulli trials, Bernoulli distribution
An experiment consists of n trials, each trial
being a success or a a failure. (Bernoulli
trial) Dene X
j
= 1 if the jth experiment
was a success and X
j
= 0 if the jth exper-
iment was a failure. The n Bernoulli trials
are called a Bernoulli process if they are in-
dependent, each trial has only two possible
outcomes (boolean trial) and the probabil-
ity of success remains constant form trial to
trial.
p(x
1
, x
2
, . . . , x
n
) = p
1
(x
1
) p
2
(x
2
) p
n
(x
n
)
where
p
j
(x
j
) =
_

_
p, x
j
= 1, j = 1, . . . , n
1 p = q , x
j
= 0, j = 1, . . . , n
0, otherwise
Bernoulli distribution.
E(X
j
) = p, V (X
j
) = pq.
17
Binomial distribution
X = # successes in n Bernoulli trials. X
has the binomial distribution given by
p(x) =
_
_
n
x
_
p
x
q
nx
, x = 0, 1, 2, . . . , n
0, otherwise
Justication of the denition of p(x): deter-
mine the probability of a particular outcome
with s initial successes (S) followed by nx
failures (F).
P(
x
..
SS . . . S
nx
..
FF . . . F) = p
x
q
nx
there are
_
n
x
_
possible combinations (out-
comes) having the required number of S

s
and F

s.
E(X) = np, V (X) = npq.
HINT: Write X = X
1
+X
2
+. . . +X
n
, where
each X
i
is an independent Bernoulli random
variable with mean p and variance pq.
Example: Binomial
computer chips are manufactured, 2% defec-
tive on the average. every day, a random
sample of 50 chips is examined. if the sam-
ple contains more than 2 defective chips, the
process will be stopped.
what is the probability that the process is
stopped by this sampling scheme?
the sampling process consists of n = 50 Bernoulli
trials, each with p = 0.02.
X = total # of defective chips in the sample.
X has a binomial distribution given by:
p(x) =
_
_
50
x
_
0.02
x
0.98
50x
, x = 0, 1, 2, . . . , 50
0, otherwise
the probability that the process is stopped
by this sampling scheme is the probability
that we more than two defective chips are
detected in the sample:
P(X > 2) = 1 P(X 2)
18
P(X 2) =
2

x=0
_
50
x
_
0.02
x
0.98
50x
= 0.92.
How much is the mean number of defective
chips in a random sample of size 50? E(X)
Geometric distribution
X = # of trials to achieve the rst suc-
cess in a sequence of Bernoulli trials. X
has the geometric distribution given by
p(x) =
_
q
x1
p, x = 1, 2, . . . ,
0, otherwise
Justication of the denition of p(x): de-
termine the probability of occurrence of
the event {X = x} (x1 failures followed
by 1 success)
P(FF . . . F
. .
x1
S) = q
x1
p
E(X) =
1
p
, Dierentiate V (X) =
q
p
2
.
Example: Geometric
40% of assembled printers are rejected during
inspection
Find the probability that the 1st acceptable
printer is the 5th one inspected.
Each inspection is a Bernoulli trial with p =
0.6 and q = 0.4.
p(5) = 0.4
4
0.6
Poisson distribution
pmf:
p(x) =
_
e

x
x!
x = 0, 1, 2, . . .
0, otherwise
with > 0.
Important Property:
E(X) = = V (X).
cdf:
F(x) =
x

i=0
e

i
i!
19
Example: Poisson
a repairperson/doctor is beeped when there
is a call for service.
# of beeps per hour, is known to follow a
Poisson distribution with a mean of = 2 per
beeps per hour.
Q1: what is the probability of 3 beeps in the
next hour?
p(3) =
e
2
2
3
3!
= F(3) F(2)
Q2: what is the probability of 2 or more
beeps in the next hour?
p(2 or more) = 1 p(0) p(1) = 1 F(1)
20
Continuous Distributions (8)
Continuous random variables are used to describe
random phenomena in which random variables
can take any value from some interval.
Uniform Distribution
X is uniformly distributed on the interval [a, b]
when the pdf is:
f(x) =
_
1
ba
, a x b
0, otherwise
cdf:
F(x) =
_

_
0, x < a
xa
ba
, a x b
1, x > b
Remark: If a x
1
< x
2
b then the probabil-
ity
P(x
1
< X < x
2
) = F(x
2
) F(x
1
) =
x
2
x1
b a
is proportional to the length of the interval
[x
1
, x
2
]
E(X) =
a +b
2
V (X) =
(b a)
2
12
21
Importance of the Uniform Distribution: lies
in the fact that random numbers uniformly
distributed in [0, 1] are used to generate sam-
ples of random variates from all other distri-
butions.
Example: Uniform A bus arrives every 20
minutes at a specic station. (Begin: 6 :
40 a.m. End: 8 : 40 a.m.) A passenger ar-
rives randomly (uniformly distributed) be-
tween 7 : 00 a.m. and 7 : 30 a.m. every
morning.
Q. What is the probability that the pas-
senger has to wait more than 5 minutes
for the bus?
The passenger has to wait more than 5
minutes for the bus only if they arrive be-
tween 7 : 00 a.m. and 7.15 a.m. or be-
tween 7 : 20 a.m. and 7 : 30 a.m.
X = # of minutes past 7 : 00 a.m. when
the passenger arrives, X is a uniform ran-
dom variable in [0, 30].
P(0 < X < 15)+P(20 < X < 30) = F(15)F(0)+F(30)F(20) =
5
6
Exponential Distribution
X is exponentially distributed with param-
eter > 0 when its pdf is:
f(x) =
_
e
x
, x 0
0, otherwise
Exp. distribution is used to model: (a)
completely random interarrival times (b)
highly variable service times (c) lifetime
of a component that fails instantaneously
(e.g. light bulb)
In these cases, the parameter is inter-
preted as a rate: arrivals/hour, services/minute,
failure rate respectively.
What is the value on the y axis where the
exp. pdf diagram intersects this axis?
Why do any two diagrams for two dier-
ent exp. pdfs (with parameters
1
and

2
) eventually intersect?
E(X) =
1

V (X) =
1

2
cdf:
F(X) =
_
1 e
x
, x 0
0, otherwise
Example: Exponential
The lifetime of a lamp in 1000s of hours,
is exp. distributed with failure rate =
1
3
(one failure every 3000 hours on the aver-
age)
Q. What is the probability that the lamp
will last longer than its expected lifetime?
P(X > E(X)) = P(X > 3) = 1P(X 3) = 1F(1) =
1
e
Remark: this result is independent of !
Q. What is the probability that the lamp
will last between 2000 hours and 3000 hours?
P(2 X 3) = F(3) F(2) =
1
e
2/3

1
e
Memoryless Property: s 0, t 0
P(X > s +t|X > s) = P(X > t)
Justication: used cdf + the denition of
the conditional probability,
P(X > s +t|X > s) =
P(X > s +t)
P(X > s)
In the lamp example, the memoryless prop-
erty means that the probability that the
lamp does not fail for at least s +t hours
given that it didnt fail for s hours, is the
same as the initial probability that it wont
fail for at least t hours.
Q. What is the probability that the lamp
will last for another 1000 hours, given that
it didnt fail for 2500 hours?
P(X > 3.5|X > 2.5) = P(X > 1) = 1F(1) =
1
e
1/3
Gamma Distribution
Gamma function: () =
_

0
x
1
e
x
dx with
> 0.
Crucial property: () = ( 1)( 1).
Generalized factorial: for integer , () =
( 1)!.
X is gamma distributed with parameters
and when its pdf is:
f(x) =
_
_
_

()
(x)
1
e
x
x > 0
0, otherwise
E(X) =
1

V (X) =
1

2
is called the shape parameter
is called the scale parameter
Note : when is an integer, gamma is
related to the exponential as follows:
If the random variable X, is the sum of
indep. exp. distributed random variables
each with parameter , then X is gamma
distributed with parameters and .
X = X
1
+X
2
+. . . +X

where each pdf of X

j
is given by:
f(x
j
) =
_
()e
x
, x 0
0, otherwise
Erlang Distribution
When = k is an integer, we refer to the
gamma distribution as the Erlang distri-
bution.
typical context: a series of k stations must
be traversed in order for a customer to
receive complete servicing. The next cus-
tomer cannot enter the rst station, until
the current customer has passed from all
the stations. Assume that each station
has an exponential distribution of service
time with parameter k.
E(X) = E(X
1
)+E(X
2
)+. . .+E(X
k
) = k
1
k
=
1

V (X) = V (X
1
)+V (X
2
)+. . .+V (X
k
) = k
1
(k)
2
=
1
k
2
Normal Distribution
X has the normal distribution with mean
( (, +)) and variance
2
if its given
by the pdf:
f(x) =
1

2
e

1
2
_
x

_
2
, < x < +
Notation: X N(,
2
) means that X is
normally distributed with mean and vari-
ance
2
.
Properties:
lim
x
f(x) = 0 & lim
x+
f(x) = 0
the values of f(x) approach 0 as x
approaches innity.
f( x) = f( + x) pdf is symmetric
about the x-axis.
maximum value (why? what is it?)
of the pdf occurs at x = . (mean =
mode)
cdf:
F(x) = P(X x) =
_
x

2
e

1
2
_
t

_
2
dt
F(x) is impossible to evaluate in closed
form.
The transformation of variables, z = (t
)/ leads to an evaluation independently
of , :
X N(,
2
), let Z =
X

,
then E(Z) = 0, V (Z) = 1.
Z N(0, 1), Z is said to have a standard
normal distribution with pdf and cdf:
(z) =
1

2
e

z
2
2
, (z) =
_
z

2
e

t
2
2
dt
The cdf (z) for the standard normal dis-
tribution has been widely tabulated.
F(x) = P(X x) = P(Z
x

) =
_
x

_
Example: Normal I
Assume X N(50, 9).
Determine F(56).
F(56) =
_
56 50
3
_
= (2) = tabulated
Intuitive Interpretation:
Example: Normal II The time required to
load a ship X is distributed as N(12, 4).
Q1. What is the probability that the ship
will be loaded in less than 10 hours?
F(10) =
_
10 12
2
_
= (1) = 1(1) = tabul.
Q2. Probability that 12 or more hours will
be required to load the ship?
P(X > 12) = 1 P(X 12) = 1 F(12) =
= 1
_
12 12
2
_
= 1(0) = 10.5 = 0.5
Q3. Probability that the time required to
load the ship will be between 10 and 12
hours?
P(10 X 12) = F(12) F(10) =
=
_
12 12
2
_

_
10 12
2
_
= (0)(1)