Basics of Probability

He Shuangchi

IE5004 Basics of Probability 1 / 42

What is probability?

Many events cannot be predicted with total certainty. The best we can say
is describing how likely they are to happen. For example, flipping a coin:
There are two possible outcomes, heads and tails.
We flip the coin many times, then we should have heads in around
one half of the flips.
We say that the probability of coin landing heads is 0.5.

Question
If we flip the coin 100 times, shall we get heads 50 times exactly?
number of heads
relative frequency of heads =
number of flips
Probability is the long-run relative frequency that an event happens.

IE5004 Basics of Probability 2 / 42

Probability as the long-run relative frequency

1 1

relative frequency
relative frequency

0.5 0.5
2/10=0.2

0 0
0 10 20 30 40 50 0 200 400 600 800 1000
number of flips number of flips

Figure: Relative frequency of heads in 1000 flips.

IE5004 Basics of Probability 3 / 42

Intuitions of probability

An experiment whose outcome cannot be known in advance is called

a random experiment.
A number between 0 and 1, called probability, is assigned to every set
of possible outcomes.
The probability of a set of possible outcomes stands for the long-run
relative frequency that the outcome is in the set when the random
experiment is repeated many times.
Probability 0 means that it is impossible that the outcome is in the
set; probability 1 means that the outcome must be in the set.
The closer the probablity is to 1/0, the more likely/unlikely the
outcome of the random experiment is in the set.

IE5004 Basics of Probability 4 / 42

Sample space

Definition
The set of all possible outcomes of a random experiment is called the
sample space. We usually denote an outcome by ω and the sample space
by Ω.

Example:
Flip a coin once, Ω = {H, T }.
Roll a die, Ω = {1, 2, 3, 4, 5, 6}.
Flip a coin twice, Ω = {HH, TH, HT , TT }.

IE5004 Basics of Probability 5 / 42

Events

Definition
An event A is a subset of the sample space, i.e., A ⊂ Ω. The event A is
said to occur if the outcome ω ∈ A.
Example:
Flip a coin once, Ω = {H, T }. Then, A = {T } is an event.
Roll a die, Ω = {1, 2, 3, 4, 5, 6}. Then,

A = {the outcome is odd} = {1, 3, 5}

is an event.
Flip a coin twice, Ω = {HH, TH, HT , TT }. Then,

A = {the first flip is heads} = {HH, HT }

is an event. If ω = HH or HT , then A occurs.

IE5004 Basics of Probability 6 / 42
 said to be mutually exclusive (or pairwise disjoint) if no two events have any out-
comes in
Basic common.
set operations
A pictorial representation of events and manipulations with events is obtained by
using Venn diagrams. To construct a Venn diagram, draw a rectangle whose interior will
represent the sample space S . Then any event A is represented as the interior of a closed
Let
curveA(often
and B be two
a circle) events.in S . Figure 2.1 shows examples of Venn diagrams.
contained

B A B A B A B
A

of (b) Shaded region (c) Shaded region (d) Shaded region (e) Mutually exclusive
B is A  B is A  B is A' events
A ∩ B (or AB) A∪B c
A =Ω\A A∩B =∅
Figure 2.1 Venn diagrams

If A ∩ B = ∅, A and B are said to be disjoint or mutually exclusive.

Let Ω = {1, 2, 3, 4, 5, 6}, A = {1, 3, 5}, B = {3, 6}. Then,
.1 (1–10)

—are participating Ain ∪ B=

a holi- {1, 3,observing
5, 6}, A ∩ = {3}, Bforc =
theB direction {1,of
each 2, 4, 5}. successive
three
he first round, 1 will play 2 vehicles.
o winners will play for the a. List all outcomes in the event A that all three vehicles go
s will also play. One possi- in the same direction.
324 (1 beats 2 and 3 beats 4 b. List all outcomes in the event B that all three vehicles take
IE5004 Basics of Probability 7 / 42
Rules of set operations

Let A, B, and C be three events.

Commutative laws

A∪B =B ∪A
A∩B =B ∩A

Associative laws

(A ∪ B) ∪ C = A ∪ (B ∪ C )
(A ∩ B) ∩ C = A ∩ (B ∩ C )

Distributive laws:

(A ∪ B) ∩ C = (A ∩ C ) ∪ (B ∩ C )
(A ∩ B) ∪ C = (A ∪ C ) ∩ (B ∪ C )

IE5004 Basics of Probability 8 / 42

DeMorgan’s laws

Let A1 , . . . , An be n events. Then,

n
[ c n
\ n
\ c n
[
Ai = Aci and Ai = Aci
i=1 i=1 i=1 i=1

Exercise
Prove DeMorgan’s law, or read the proof on page 27.

For two events A and B,

(A ∪ B)c = Ac ∩ B c and (A ∩ B)c = Ac ∪ B c

IE5004 Basics of Probability 9 / 42

Probability space

A probability space consists of three parts (Ω, F, P)

Ω is the sample space
F is the collection of events
P is the rule to assign probabilities to the events, such that
1. P[Ω] = 1
2. P[A] ≥ 0 for each event A, (i.e., for each A ∈ F)
3. If A1 , A2 , . . . are pairwise disjoint events, then
∞
h[ i X∞
P Ai = P[Ai ]
i=1 i=1

Remark:
P[A] is the probability that A occurs (or the outcome is in A).
In the rolling-a-die example, F = {∅, {1}, {2}, . . . , {1, 2}, . . . , Ω}.

IE5004 Basics of Probability 10 / 42

Properties of probability

Proposition
P[∅] = 0
S∞
Let A1 = Ω and A2 = A3 = · · · = ∅. Because Ω = i=1 Ai and the events
are disjoint, then
∞
X ∞
X
P[Ω] = P[Ai ] = P[Ω] + P[Ai ],
i=1 i=2

which implies that P[∅] = 0.

IE5004 Basics of Probability 11 / 42

Properties of probability

Proposition
If A1 , . . . , An are pairwise disjoint events, then
n
h[ i Xn
P Ai = P[Ai ]
i=1 i=1

Let An+1 = An+2 = · · · = ∅. Then,

n
h[ i ∞
h[ i X∞ n
X ∞
X n
X
P Ai = P Ai = P[Ai ] = P[Ai ] + P[Ai ] = P[Ai ]
i=1 i=1 i=1 i=1 i=n+1 i=1

IE5004 Basics of Probability 12 / 42

Properties of probability

Proposition
P[Ac ] = 1 − P[A]

Because A ∪ Ac = Ω and A ∩ Ac = ∅, then

P[A] + P[Ac ] = P[Ω] = 1

Hence, P[Ac ] = 1 − P[A].

IE5004 Basics of Probability 13 / 42

Properties of probability

Proposition
P[A] ≤ 1

This is because P[A] = 1 − P[Ac ] ≤ 1.

Proposition
If A ⊂ B, then P[A] ≤ P[B].

Since A ⊂ B, we have B = A ∪ (Ac ∩ B). Since A and Ac ∩ B are disjoint,

P[B] = P[A] + P[Ac ∩ B] ≥ P[A]

IE5004 Basics of Probability 14 / 42

 Properties of probability
oof Note first that A ´ B can be decomposed into two disjoint events, A a
¨ Ar; the latter is the part of B that lies outside A (see Figure 2.4). Furthermore,
elf is the union of the two disjoint events A ¨ B and Ar ¨ B,
B) 5 P(A ¨ B) 1 P(Ar ¨ B). Thus
Proposition
P(A ´ B) 5 P(A)
P[A 1 P(B
∪ B] =¨ Ar)+5P[B]
P[A] P(A) 1 [P(B)
− P[A ∩ B] 2 P(A ¨ B)]
5 P(A) 1 P(B) 2 P(A ¨ B)

A B
 

Figure 2.4 Representing A  B as a union of disjoint events

a certain residential suburb, 60% of all households get Internet service from t
al cable company, 80% get television service from that company, and 50% g
h services from that company. If a household is randomly selected, what is t
IE5004 Basics of Probability 15 / 42
Examples

Roll a fair die. Let A = {1, 3, 5}, B = {3, 6}, and C = {2, 4, 6}.
P[A] = 3/6
P[B] = 2/6
P[B c ] = 4/6
P[A ∪ B] = 4/6 and P[A ∩ B] = 1/6
P[A ∪ C ] = 6/6 and P[A ∩ C ] = 0

IE5004 Basics of Probability 16 / 42

Example

Roll a fair die in a gamble. Suppose that if the outcome is either 1 or 3,

you win two dollars; otherwise, you lose one dollar. Let

A = {you win} = {1, 3}.

Then,
P[A] = 1/3.
Suppose that the die was tossed but you has not seen the outcome. You
are told “the outcome is odd.” Then, what is the probability that you has
won?

IE5004 Basics of Probability 17 / 42

 Probability with “extra information”
ion (2.2), the conditional probability is expressed as a ra
bilities: The numerator is the probability of the intersect
as Let
theA denominator
and B be two events. Suppose that you are interested in the
is the probability of the conditionin
probability that A occurs. Then, the experiment is performed and you are
illuminates thisbefore
told that B occurs relationship (Figure
you can see the 2.8).
outcome. Then, with this “extra
information”, the probability that A occurs will change.

igure 2.8 Motivating the definition of conditional probability

IE5004 Basics of Probability 18 / 42

Conditional probability

Definition
For any events A and B with P[B] > 0, the conditional probability of A
given B has occurred is defined by

P[A ∩ B]
P[A|B] = .
P[B]

In the previous example,

A = {you win} = {1, 3}
B = {the outcome is odd} = {1, 3, 5}
Then,
P[A ∩ B] P[{1, 3}] 2/6 2
P[A|B] = = = =
P[B] P[{1, 3, 5}] 3/6 3

IE5004 Basics of Probability 19 / 42

Example

Flip a fair coin twice. Then, Ω = {HH, HT , TH, TT }. Let

A = {the first flip is heads} = {HH, HT }
B = {the second flip is heads} = {HH, TH}
Then, A ∩ B = {HH} and

P[A ∩ B] P[{HH}] 1/4 1

P[A|B] = = = =
P[B] P[{HH, TH}] 1/2 2

IE5004 Basics of Probability 20 / 42

Example

Exercise
A bin contains 5 defective (that immediately fail when put in use), 10
partially defective (that fails after a couple of hours of use), and 25
acceptable transistors. A transistor is chosen at random from the bin and
put into use. If it does not immediately fail, what is the probability that it
is acceptable?

Since the transistor did not fail immediately, it is not one of the 5
defectives. So the desired probability is

P[acceptable, not defective]

P[acceptable|not defective] =
P[not defective]
P[acceptable] 25/40 5
= = =
P[not defective] 35/40 7

IE5004 Basics of Probability 21 / 42

Multiplication rule

Proposition
P[A ∩ B] = P[A|B]P[B].

Exercise
Ms. Lee figures that there is a 30% chance that her company will set up a
branch office in Singapore. If it does, she is 60% certain that she will be
made manager of this new operation. What is the probability that she will
be a Singapore branch office manager?

Let B = {the company sets up the Singapore branch} and A = {she is

made the Singapore manager}. Then,

P[A] = P[A ∩ B] = P[A|B]P[B] = 0.6 × 0.3 = 0.18

IE5004 Basics of Probability 22 / 42

Tree diagram

P[A∩B]=0.18

IE5004 Basics of Probability 23 / 42

Multiplication rule

Proposition

P[E1 ∩ E2 ∩ · · · ∩ En ] = P[E1 ]P[E2 |E1 ]P[E3 |E1 ∩ E2 ] · · · P[En |E1 ∩ · · · ∩ En−1 ]

It follows from the fact

P[E1 ∩ E2 ] P[E1 ∩ E2 ∩ E3 ] P[E1 ∩ · · · ∩ En ]
P[E1 ∩E2 ∩· · ·∩En ] = P[E1 ] ···
P[E1 ] P[E1 ∩ E2 ] P[E1 ∩ · · · ∩ En−1 ]

IE5004 Basics of Probability 24 / 42

Proof Because the Ai’s are mutually exclusive and exhaustive, if B occurs it must b
Law of total probability
n conjunction with exactly one of the Ai’s. That is, B 5 (A1 ¨ B) ´ c ´ (Ak ¨ B
where the events (Ai ¨ B) are mutually exclusive. This “partitioning of B” is illustrate
Proposition
n Figure 2.11. Thus
Let A1 , . . . , Ak be disjoint events
k and A1 ∪ k· · · ∪ Ak = Ω. Then, for any
event B, P(B) 5 g P(Ai ¨ B) 5 g P(Bu Ai)P(Ai)
i51 1 ]P[A1 ] + · · ·i51
P[B] = P[B|A + P[B|Ak ]P[Ak ].
s desired.
B

A1 A3

A4
A2

Figure 2.11 Partition of B by mutually exclusiveS

and exhaustive Ai’s ■
k
Because A1 ∩ B, . . . , Ak ∩ B are disjoint and B = i=1 (Ai ∩ B), then
An P[B]
individual
= P[Ahas 3 different email accounts. Most of her messages, in fact 70%
1 ∩ B] + · · · + P[Ak ∩ B] = P[B|A1 ]P[A1 ] + · · · + P[B|Ak ]P[Ak ]
come into account #1, whereas 20% come into account #2 and the remaining 10%
nto accountIE5004
#3. Of the messages Basics
into ofaccount #1, only 1% are spam, whereas
Probability 25 / 42th
Example

Exercise
An insurance company believes that people can be divided into two
classes—those that are accident prone and those that are not. Their
statistics show that an accident-prone person will have an accident at
some time within a fixed 1-year period with probability 0.4, whereas this
probability decreases to 0.2 for a non-accident-prone person. If we assume
that 30 percent of the population is accident prone, what is the probability
that a new policy holder will have an accident within a year of purchasing
a policy?

Let A = {the policy holder is accident-prone} and B = {the policy holder

will have an accident}.

P[B] = P[A ∩ B] + P[Ac ∩ B] = P[B|A]P[A] + P[B|Ac ]P[Ac ]

= 0.4 × 0.3 + 0.2 × 0.7 = 0.26

IE5004 Basics of Probability 26 / 42

Tree diagram

P[A∩B]=0.12

P[A’∩B]=0.14

IE5004 Basics of Probability 27 / 42

Bayes’ formula

Proposition
Let A1 , . . . , Ak be disjoint events and A1 ∪ · · · ∪ Ak = Ω. Then, for any
event B with P[B] > 0 and any j = 1, . . . , k,

P[Aj ∩ B] P[B|Aj ]P[Aj ]

P[Aj |B] = =
P[B] P[B|A1 ]P[A1 ] + · · · + P[B|Ak ]P[Ak ]

IE5004 Basics of Probability 28 / 42

Example

Exercise
A plane is missing and it is presumed that it was equally likely to have gone
down in any of three possible regions. Let 1 − αi denote the probability
the plane will be found upon a search of the ith region when the plane is,
in fact, in that region. What is the conditional probability that the plane is
in the ith region, given that a search of region 1 is unsuccessful?

Let Ri = {plane in region i} and E = {unsuccessful search of region 1}.

α1
P[R1 ∩ E ] P[E |R1 ]P[R1 ] 3 α1
P[R1 |E ] = = P3 = α1 1 1
= .
P[E ] i=1 P[E |Ri ]P[Ri ] 3 + 3 + 3
α1 + 2

For i = 2, 3,
1
P[Ri ∩ E ] P[E |Ri ]P[Ri ] 3 1
P[Ri |E ] = = P3 = α1 1 1
= .
P[E ] i=1 P[E |Ri ]P[Ri ] 3 + 3 + 3
α1 + 2

IE5004 Basics of Probability 29 / 42

Tree diagram
P[E∩R1]= α1/3

P[E∩R2]= 1/3

P[E∩R3]= 1/3

IE5004 Basics of Probability 30 / 42

Example

Exercise
Only 1 in 1000 adults is afflicted with a rare disease for which a diagnostic
test has been developed. When an individual has the disease, a positive
test result will occur 99% of the time, whereas an individual without the
disease will show a positive test result only 2% of the time. If a randomly
selected individual is tested and the result is positive, what is the
probability that the individual has the disease?

IE5004 Basics of Probability 31 / 42

 1 2
does not have the disease, and B 5 positive test result. Then P(A1) 5 .001,
Tree
P(A2)diagram
5 .999, P(Bu A1) 5 .99, and P(Bu A2) 5 .02. The tree diagram for this prob-
lem is in Figure 2.12.
P(A1  B)  .00099
.99
st
Te
B
.01
.001
se B' 
d isea  Test
Has
A1 
.999 P(A2  B)  .01998
A2  .02
Doe
sn' st
t ha T e
ve d
i seas B
e .98
B' 
 Test

Figure 2.12 Tree diagram for the rare-disease problem

P[B] = P[A1 ∩ B] + P[A2 ∩ B] = 0.00099 + 0.01998 = 0.02097,

ed, scanned, or duplicated, in whole or in part. Due to electronic rights, some third party content may be suppressed from the eBook and/or eChapter(s).
P[A
affect the overall learning experience. 1 ∩Learning
Cengage B] reserves the right to remove additional content at any time if subsequent rights restrictions require it.
P[A1 |B] = = 0.00099/0.02097 = 0.04721.
P[B]

IE5004 Basics of Probability 32 / 42

Independence

Let A and B be two events.

In general, knowing “B occurs” would change the chance “A occurs”.
That is, P[A] is changed to P[A|B].
It is possible that B “has nothing to do” with A, i.e., P[A|B] = P[A].

IE5004 Basics of Probability 33 / 42

Example

Flip a fair coin twice. Then, Ω = {HH, HT , TH, TT }. Let

A = {the first flip is heads} = {HH, HT }
B = {the second flip is heads} = {HH, TH}
Then, A ∩ B = {HH} and

P[A ∩ B] P[{HH}] 1/4 1

P[A|B] = = = =
P[B] P[{HH, TH}] 1/2 2

Since
P[A] = P[{HH, HT }] = 1/2 = P[A|B],
B “has nothing to do” with A.

IE5004 Basics of Probability 34 / 42

Independence

Definition
Two events A and B are independent if P[A|B] = P[A].

If A and B are independent, then P[A ∩ B] = P[A|B]P[B] = P[A]P[B].

If P[A ∩ B] = P[A]P[B], then

P[A ∩ B] P[A]P[B]
P[A|B] = = = P[A].
P[B] P[B]

Proposition
Two events A and B are independent if and only if

P[A ∩ B] = P[A]P[B].

IE5004 Basics of Probability 35 / 42

Example

A card is selected at random from an ordinary deck of 52 playing cards. If

A is the event that the selected card is an ace and H is the event that it is
a heart, then,
P[A ∩ H] = P[{the card is the ace of hearts}] = 1/52
P[A] = 4/52 = 1/13
P[H] = 13/52 = 1/4
Because P[A ∩ H] = P[A]P[H], A and H are independent.

IE5004 Basics of Probability 36 / 42

Independence and complementation

Proposition
If A and B are independent, then so are A and B c .

Since A = (A ∩ B) ∪ (A ∩ B c ) and A ∩ B and A ∩ B c are disjoint, then

P[A] = P[A ∩ B] + P[A ∩ B c ] = P[A]P[B] + P[A ∩ B c ]

It follows that

P[A ∩ B c ] = P[A] − P[A]P[B] = P[A](1 − P[B]) = P[A]P[B c ]

Question
If A and B are independent, are Ac and B c independent?

IE5004 Basics of Probability 37 / 42

Independence of three events

Definition
Three events A, B, and C are said to be independent if

P[A ∩ B ∩ C ] = P[A]P[B]P[C ]
P[A ∩ B] = P[A]P[B]
P[B ∩ C ] = P[B]P[C ]
P[C ∩ A] = P[C ]P[A]

IE5004 Basics of Probability 38 / 42

Example

Exercise
A system composed of n separate components is said to be a parallel
system if it functions when at least one of the components functions. For
such a system, if component i, independent of other components,
functions with probability pi , i = 1, . . . , n, what is the probability the
system functions?
3.8 Independent Events 79

2
A B
3

FIGURE 3.7 Parallel system: functions if current flows from A to B.

It is sometimes the case that the probability experiment under consideration consists of
performing
IE5004 a sequence of subexperiments. For instance, if the experiment consists
Basics of Probability 39 /of
42
Solution

Let Ai = {the component i functions}. Then, P[Ai ] = pi .

P[{system functions}] = 1 − P[{system does not function}]

= 1 − P[{no components function}]
= 1 − P[Ac1 ∩ · · · ∩ Acn ]
= 1 − P[Ac1 ] · · · P[Acn ]
= 1 − (1 − p1 ) · · · (1 − pn ) by independence.

Suppose n = 6 and p1 = · · · = p6 = 0.5, then

P[{system functions}] = 1 − 0.56 = 0.9844

IE5004 Basics of Probability 40 / 42

 illuminates this relationship (Figure 2.8).
P[·|B] is a probability

igure 2.8 Motivating the definition of conditional probability

Proposition
1 P[Ω|B] = 1
2 P[A|B] ≥ 0 for each event A
P ∞ = ∞
S 
at B has
If A1 , occurred, the disjoint,
relevant thensample space is P[A
noi |B]
long
P
A2 , . . . are pairwise i=1 Ai |B
3
i=1

mesExercise
in B; A has occurred if and only if one of the outcome
ed,Prove
so the
theaboveconditional
proposition. probability of A given B is pr
proportionality constant 1/P(B) is used to ensure that t
IE5004 Basics of Probability 41 / 42
Homework

Reading assignment
Study Chapters 1.1–1.4 on your own
Read Chapters 2.1–2.5
Read Chapters 3.1–3.5
Exercise problems
1.1, 1.10, 1.19, 1.21, 1.22
2.10, 2.18, 2.20, 2.23
3.1, 3.17, 3.43, 3.49, 3.52

IE5004 Basics of Probability 42 / 42