Probability &

Statistics

Conditional
Probabili- Probability & Statistics
ties
Basic Probability 2
Independence

Counting
TRAN Hoang Tung
NGUYEN Minh Huong
NGUYEN Mai Chung

University of Science and Technology of Hanoi (USTH)

September 10, 2019

1/18
 The Course So Far
Probability &
Statistics
What we have learnt?

Conditional
Probabili-
ties

Independence

Counting

2/18
 The Course So Far
Probability &
Statistics
What we have learnt?

Conditional Probability model: sample space, axioms

Probabili-
ties

Independence

Counting

2/18
 The Course So Far
Probability &
Statistics
What we have learnt?

Conditional Probability model: sample space, axioms

Probabili-
ties Notations are very important!
Independence

Counting

2/18
 The Course So Far
Probability &
Statistics
What we have learnt?

Conditional Probability model: sample space, axioms

Probabili-
ties Notations are very important!
Independence

Counting
Today

2/18
 The Course So Far
Probability &
Statistics
What we have learnt?

Conditional Probability model: sample space, axioms

Probabili-
ties Notations are very important!
Independence

Counting
Today

Conditional probability, total probability theorem, Bayes’

rule

2/18
 The Course So Far
Probability &
Statistics
What we have learnt?

Conditional Probability model: sample space, axioms

Probabili-
ties Notations are very important!
Independence

Counting
Today

Conditional probability, total probability theorem, Bayes’

rule
Independence, conditional independence

2/18
 The Course So Far
Probability &
Statistics
What we have learnt?

Conditional Probability model: sample space, axioms

Probabili-
ties Notations are very important!
Independence

Counting
Today

Conditional probability, total probability theorem, Bayes’

rule
Independence, conditional independence
Counting

2/18
 Probability &
Statistics

1 Conditional Probabilities
Conditional Definition
Probabili- Total Probability Theorem
ties
Bayes’ Rule
Independence

Counting
2 Independence
Definition
Conditional Independence

3 Counting
Counting Principle
Combinations

3/18
 Probability &
Statistics
1 Conditional Probabilities
Definition
Conditional
Probabili- Total Probability Theorem
ties Bayes’ Rule
Definition
Total
Probability
Theorem 2 Independence
Bayes’ Rule

Independence 3 Counting
Counting

4/18
 Introduction
Probability &
Statistics Consider 2 experiments:

Conditional
Probabili-
ties
Definition
Total
Probability
Theorem
Bayes’ Rule

Independence

Counting

5/18
 Introduction
Probability &
Statistics Consider 2 experiments:
Roll a fair 6-sided die
Conditional Sample space Ω =
Probabili- Probability P (1) = P (2) =
ties
Definition
Total
Probability
Theorem
Bayes’ Rule

Independence

Counting

5/18
 Introduction
Probability &
Statistics Consider 2 experiments:
Roll a fair 6-sided die
Conditional Sample space Ω = {1, 2, 3, 4, 5, 6}
Probabili- Probability P (1) = 1/6 P (2) = 1/6
ties
Definition
Total
Probability
Theorem
Bayes’ Rule

Independence

Counting

5/18
 Introduction
Probability &
Statistics Consider 2 experiments:
Roll a fair 6-sided die
Conditional Sample space Ω = {1, 2, 3, 4, 5, 6}
Probabili- Probability P (1) = 1/6 P (2) = 1/6
ties Roll a fair 6-sided die, knowing the outcome is even
Definition
Total
Probability
Theorem
Bayes’ Rule

Independence

Counting

5/18
 Introduction
Probability &
Statistics Consider 2 experiments:
Roll a fair 6-sided die
Conditional Sample space Ω = {1, 2, 3, 4, 5, 6}
Probabili- Probability P (1) = 1/6 P (2) = 1/6
ties Roll a fair 6-sided die, knowing the outcome is even
Definition
Event A = {2, 4, 6}
Total
Probability Probability P (1 | A) = P (2 | A) =
Theorem
Bayes’ Rule

Independence

Counting

5/18
 Introduction
Probability &
Statistics Consider 2 experiments:
Roll a fair 6-sided die
Conditional Sample space Ω = {1, 2, 3, 4, 5, 6}
Probabili- Probability P (1) = 1/6 P (2) = 1/6
ties Roll a fair 6-sided die, knowing the outcome is even
Definition
Event A = {2, 4, 6}
Total
Probability Probability P (1 | A) = 0 P (2 | A) = 1/3
Theorem
Bayes’ Rule

Independence

Counting

5/18
 Introduction
Probability &
Statistics Consider 2 experiments:
Roll a fair 6-sided die
Conditional Sample space Ω = {1, 2, 3, 4, 5, 6}
Probabili- Probability P (1) = 1/6 P (2) = 1/6
ties Roll a fair 6-sided die, knowing the outcome is even
Definition
Event A = {2, 4, 6}
Total
Probability Probability P (1 | A) = 0 P (2 | A) = 1/3
Theorem
Bayes’ Rule

Independence
Conditional Probabilities say “Hello USTH Students!!!”
Counting

5/18
 Definition
Probability &
Statistics

Conditional
Probabili-
ties
Definition
Total
Probability
Theorem
Bayes’ Rule

Independence

Counting

6/18
 Definition
Probability &
Statistics

Conditional
Probabili-
ties
Definition
Total
Probability
Theorem Conditional Probability
Bayes’ Rule

Independence P (A ∩ B)
P (A | B) ,
Counting P (B)

6/18
 Definition
Probability &
Statistics

Conditional
Probabili-
ties
Definition
Total
Probability
Theorem Conditional Probability
Bayes’ Rule

Independence P (A ∩ B)
P (A | B) ,
Counting P (B)

Multiplication Rule

P (A ∩ B) = P (B)P (A | B) = P (A)P (B | A)

6/18
 Total Probability Theorem
Probability &
Statistics

Conditional
Probabili-
ties
Definition
Total
Probability
Theorem
Bayes’ Rule

Independence

Counting

7/18
 Total Probability Theorem
Probability &
Statistics
Partition into A1 , A2 , A3
Conditional
Probabili-
ties
Definition
Total
Probability
Theorem
Bayes’ Rule

Independence

Counting

7/18
 Total Probability Theorem
Probability &
Statistics
Partition into A1 , A2 , A3
Conditional Have P (Ai ) for every i
Probabili-
ties
Definition
Total
Probability
Theorem
Bayes’ Rule

Independence

Counting

7/18
 Total Probability Theorem
Probability &
Statistics
Partition into A1 , A2 , A3
Conditional Have P (Ai ) for every i
Probabili-
ties
Have P (B | Ai ) for every i
Definition
Total
Probability
Theorem
Bayes’ Rule

Independence

Counting

7/18
 Total Probability Theorem
Probability &
Statistics
Partition into A1 , A2 , A3
Conditional Have P (Ai ) for every i
Probabili-
ties
Have P (B | Ai ) for every i
Definition P (B) = ?
Total
Probability
Theorem
Bayes’ Rule

Independence

Counting

7/18
 Total Probability Theorem
Probability &
Statistics
Partition into A1 , A2 , A3
Conditional Have P (Ai ) for every i
Probabili-
ties
Have P (B | Ai ) for every i
Definition P (B) = ?
Total
Probability
Theorem Definition
Bayes’ Rule
X
Independence P (B) = P (Ai )P (B | Ai )
Counting i

7/18
 Bayes’ Rule
Probability &
Statistics
Partition into A1 , A2 , A3
Conditional Have P (Ai ) for every i
Probabili-
ties
Have P (B | Ai ) for every i
Definition
Total
Probability
Theorem
Bayes’ Rule

Independence

Counting

8/18
 Bayes’ Rule
Probability &
Statistics
Partition into A1 , A2 , A3
Conditional Have P (Ai ) for every i initial “beliefs”
Probabili-
ties
Have P (B | Ai ) for every i
Definition
Total
Probability
Theorem
Bayes’ Rule

Independence

Counting

8/18
 Bayes’ Rule
Probability &
Statistics
Partition into A1 , A2 , A3
Conditional Have P (Ai ) for every i initial “beliefs”
Probabili-
ties
Have P (B | Ai ) for every i
Definition revised “beliefs”, given B occurred
Total
Probability
Theorem
Bayes’ Rule

Independence

Counting

8/18
 Bayes’ Rule
Probability &
Statistics
Partition into A1 , A2 , A3
Conditional Have P (Ai ) for every i initial “beliefs”
Probabili-
ties
Have P (B | Ai ) for every i
Definition revised “beliefs”, given B occurred
Total multiplication rule
Probability P (Ai | B) =
Theorem
total probability theorem
Bayes’ Rule

Independence

Counting

8/18
 Bayes’ Rule
Probability &
Statistics
Partition into A1 , A2 , A3
Conditional Have P (Ai ) for every i initial “beliefs”
Probabili-
ties
Have P (B | Ai ) for every i
Definition revised “beliefs”, given B occurred
Total
Probability
Theorem Bayes’ Rule
Bayes’ Rule

Independence P (Ai )P (B | Ai )
P (Ai | B) = P
Counting j P (Aj )P (B | Aj )

8/18
 Bayesian Inference - A Flash Introduction
Probability &
Statistics
Initial beliefs P (Ai ) on possible causes of an observed
event B
Conditional
Probabili-
ties
Definition
Total
Probability
Theorem
Bayes’ Rule

Independence

Counting

9/18
 Bayesian Inference - A Flash Introduction
Probability &
Statistics
Initial beliefs P (Ai ) on possible causes of an observed
event B
Conditional
Probabili- Model of the world under each Ai
ties
Definition model
Total
Ai −−−−−−→ B
P (B | Ai )
Probability
Theorem
Bayes’ Rule

Independence

Counting

9/18
 Bayesian Inference - A Flash Introduction
Probability &
Statistics
Initial beliefs P (Ai ) on possible causes of an observed
event B
Conditional
Probabili- Model of the world under each Ai
ties
Definition model
Total
Ai −−−−−−→ B
P (B | Ai )
Probability
Theorem
Bayes’ Rule Draw inference about causes
Independence inference
B −−−−−−→ Ai
Counting P (Ai | B)

9/18
 Probability &
Statistics
1 Conditional Probabilities
Conditional
Probabili- 2 Independence
ties
Definition
Independence Conditional Independence
Definition
Conditional
Independence
3 Counting
Counting

10/18
 Independence
Probability &
Statistics
Intuition explanation
Conditional P (roll a fair die and get 6 | it is raining) = . . .
Probabili-
ties

Independence
Definition
Conditional
Independence

Counting

11/18
 Independence
Probability &
Statistics
Intuition explanation
Conditional P (roll a fair die and get 6 | it is raining) = . . .
Probabili-
ties
Intuition “definition”
Independence
Definition
Conditional
P (B | A) = P (B)
Independence

Counting

11/18
 Independence
Probability &
Statistics
Intuition explanation
Conditional P (roll a fair die and get 6 | it is raining) = . . .
Probabili-
ties
Intuition “definition”
Independence
Definition
Conditional
P (B | A) = P (B)
Independence

Counting Definition of independence

P (A ∩ B) = P (A)P (B)

11/18
 Independence
Probability &
Statistics
Intuition explanation
Conditional P (roll a fair die and get 6 | it is raining) = . . .
Probabili-
ties
Intuition “definition”
Independence
Definition
Conditional
P (B | A) = P (B)
Independence

Counting Definition of independence

P (A ∩ B) = P (A)P (B)

Quick check: two disjoint events, independent?

11/18
 Independence
Probability &
Statistics
Intuition explanation
Conditional P (roll a fair die and get 6 | it is raining) = . . .
Probabili-
ties
Intuition “definition”
Independence
Definition
Conditional
P (B | A) = P (B)
Independence

Counting Definition of independence

P (A ∩ B) = P (A)P (B)

Quick check: two disjoint events, independent?

A and B are independent, what about A and B c ?

11/18
 Conditional Independence
Probability &
Statistics

Conditional
Probabili-
ties

Independence
Definition
Conditional
Independence

Counting

12/18
 Conditional Independence
Probability &
Statistics

Conditional
Probabili-
ties

Independence
Definition
Conditional
Independence

Counting

12/18
 Conditional Independence
Probability &
Statistics

Conditional
Probabili-
ties

Independence
Definition
Conditional
Definition
Independence
Conditional independence, given C
Counting
P (A ∩ B | C) = P (A | C)P (B | C)

12/18
 Conditional Independence
Probability &
Statistics

Conditional
Probabili-
ties

Independence
Definition
Conditional
Definition
Independence
Conditional independence, given C
Counting
P (A ∩ B | C) = P (A | C)P (B | C)

Quiz: Does independence imply conditional independence?

12/18
 Probability &
Statistics

Conditional
Probabili-
ties

Independence

Counting
5-min Break!

13/18
 Probability &
Statistics
1 Conditional Probabilities
Conditional
Probabili- 2 Independence
ties

Independence
3 Counting
Counting Counting Principle
Counting
Principle
Combinations
Combinations

14/18
 Discrete Uniform Law
Probability &
Statistics
Ω consists of n equally likely elements
Conditional A consists of k elements
Probabili-
ties

Independence

Counting Then
number of elements of A k
Counting P (A) = =
Principle number of elements of Ω n
Combinations

15/18
 Counting Principle
Probability &
Statistics
Example

Conditional 4 shirts
Probabili-
ties 3 ties
Independence 2 jackets
Counting In how many ways can you get dressed?
Counting
Principle
Combinations

16/18
 Counting Principle
Probability &
Statistics
Example

Conditional 4 shirts
Probabili-
ties 3 ties
Independence 2 jackets
Counting In how many ways can you get dressed?
Counting
Principle
Combinations Counting Principle
Consider a sequence of r stages, each stage has a number of
options ni . Then the number of possible choices is n1 n2 . . . nr

16/18
 Examples
Probability &
Statistics
License Plates
Consider license plates that consist of 2 letters followed by 3
Conditional
Probabili- digits. How many different license plates are there?
ties

Independence

Counting
Counting
Principle
Combinations

17/18
 Examples
Probability &
Statistics
License Plates
Consider license plates that consist of 2 letters followed by 3
Conditional
Probabili- digits. How many different license plates are there?
ties
if there is no repetition
Independence

Counting
Counting
Principle
Combinations

17/18
 Examples
Probability &
Statistics
License Plates
Consider license plates that consist of 2 letters followed by 3
Conditional
Probabili- digits. How many different license plates are there?
ties
if there is no repetition
Independence

Counting Permutations
Counting
Principle Number of ways of ordering n elements:
Combinations

17/18
 Examples
Probability &
Statistics
License Plates
Consider license plates that consist of 2 letters followed by 3
Conditional
Probabili- digits. How many different license plates are there?
ties
if there is no repetition
Independence

Counting Permutations
Counting
Principle Number of ways of ordering n elements: n!
Combinations

17/18
 Examples
Probability &
Statistics
License Plates
Consider license plates that consist of 2 letters followed by 3
Conditional
Probabili- digits. How many different license plates are there?
ties
if there is no repetition
Independence

Counting Permutations
Counting
Principle Number of ways of ordering n elements: n!
Combinations

Subsets
Number of subsets of {1, 2, . . . , n}:

17/18
 Examples
Probability &
Statistics
License Plates
Consider license plates that consist of 2 letters followed by 3
Conditional
Probabili- digits. How many different license plates are there?
ties
if there is no repetition
Independence

Counting Permutations
Counting
Principle Number of ways of ordering n elements: n!
Combinations

Subsets
Number of subsets of {1, 2, . . . , n}: 2n

17/18
 Combinations
Probability &
Statistics
Problem
Given a set of n elements. We want to construct a subset
Conditional
Probabili- with exactly k elements. In how many ways can this be done?
ties

Independence

Counting
Counting
Principle
Combinations

18/18