Information Theory and Coding (EC60332) Spring 2023

Lecture 1: Probability Review

Instructor: Jithin Ravi Scribe: C. Ajay Kumar (9 Jan. 2023)

In this lecture, we brush up our knowledge on probability and refresh some of the concepts which
will be required as we move forward in our course of “ Information Theory and Coding ”.

1 Probability Space (Ω, F ,P)

In order to comprehend a statement like “ the probability of rolling consecutive sixes while rolling a
die twice is about 3% ”, we need to specify a probability space. A probability space models random
experiments and consists of three components:

O1 Sample space (Ω): The set of all possible outcomes of a random experiment. For example,

O
Tossing a coin: Ω = {H, T }, Throwing a die: Ω = {1, 2, 3, 4, 5, 6}.
2 σ-algebra (F ): Set of events or we can say a collection of subsets of sample space. This
collection of subsets is a σ-field if it satisfies the following conditions:
(i) Ω ϵ F .
(ii) If A ϵ F, then Ac ϵ F .
n
 n [
(iii) If Ai i=1 ϵ F , then Ai ϵ F . Here, n can be countably infinite.
i=1
Based on the above conditions, the min. σ-algebra is F = {ϕ, Ω} and the most generally considered
σ-algebra is Power set (2|Ω| ).
O3 Probability measure (P): It is a function P: F −→ [0, 1], which assigns a probability to
each event and follows the axioms of probability.
Example: For an experiment of tossing a coin.
Ω = {H, T }.

F = ϕ, {H}, {T }, Ω −→ Power set.

1
1
P {H} = , P {T } = , P(ϕ) = 0, P(Ω) = 1.
2 2

1.1 Axioms of Probability

O
O
1 (Normalization) The probability of the sample space(Ω) is equal to 1, i.e. P(Ω) = 1.

O
2 (Non-negativity) For every event A ϵ F, P(A) ≥ 0.
3 (Additivity) If A, B are disjoint events, then the probability of their union satisfies
P(A ∪ B) = P(A) + P(B).
More generally, if the sample space consists of countably infinite number of elements and A1 ,
A2 , . . . is a sequence of disjoint events, then the probability of their union satisfies
P(A1 ∪ A2 ∪ · · · ) = P(A1 ) + P(A2 ) + · · · .

1
1.2 Union Bound
For any two events A and B we have
P(A ∪ B) = P(A) + P(B) − P(A ∩ B)
≤ P(A) + P(B)
Similarly, for three events A, B and C, we can write
P(A ∪ B ∪ C) = P((A ∪ B) ∪ C)
≤ P(A ∪ B) + P(C)
≤ P(A) + P(B) + P(C)
In general, for any events A1 , A2 , . . . , An , we have
P(A1 ∪ A2 ∪ · · · ∪ An ) ≤ P(A1 ) + P(A2 ) + · · · + P(An )
n n
!
[ X
⇒P Ai ≤ P(Ai )
i=1 i=1
This is known as union bound or Boole’s inequality.

1.3 Conditional Probability

Conditional probability provides us with a way to reason about the outcome of an experiment based
on partial information. In more precise terms, given an experiment, a corresponding sample space,
and a probability measure, suppose that we know that the outcome is within some given event B;
Then, P(A|B) is a new probability measure that takes into account the available knowledge and
specifies the conditional probability of A given B.
P(A ∩ B)
P(A|B) = , where P(B) > 0.
P(B)
Example: Three Coins. Each of them can be chosen with equal probability.
C1 : H, H C2 : H, T C3 : T, T
1
P(C1 ) = P(C2 ) = P(C3 ) =
3
Then,
P(C1 ) · P(H|C1 )
P(C1 |H) =
P(C1 ) · P(H|C1 ) + P(C2 ) · P(H|C2 ) + P(C3 ) · P(H|C3 )
1
·1
= 3
1 1 1 1
·1+ · + ·0
3 3 2 3
2
=
3

1.4 Bayes’ Theorem

Let A1 , A2 , . . . , An be disjoint events that form a partition of the sample space, and assume that
P(Ai ) > 0, ∀ i. Then, for any event B such that P(B) > 0, we have

2
 P(Ai ) · P(B|Ai )
P(Ai |B) =
P(B)

P(Ai ) · P(B|Ai )
=
P(A1 ) · P(B|A1 ) + · · · + P(An ) · P(B|An )

P(Ai ) · P(B|Ai )
= n
P
P(Ai ) · P(B|Ai )
i=1
P(Ai |B) is referred to as the a posteriori probability of Ai given the information, while P (Ai ) is
referred to as the priori probability.

1.5 Total Probability Law

Let A1 , . . . , An be disjoint events that form a partition of the sample space (each possible outcome
is included in exactly one of the events A1 , . . . , An ) and assume that P(Ai ) > 0 ∀ i. Then, for any
event B, we have
P(B) = P(A1 ∩ B) + · · · + P(An ∩ B)
= P(A1 ) · P(B|A1 ) + · · · + P(An ) · P(B|An )
n
X n
[
= P(Ai ) · P(B|Ai ) where, Ai = Ω
i=1 i=1

2 Random Variables
Random variable X is a real-valued function of the experimental outcome i.e. it is a mapping from
sample space(Ω) to a real line R.
X : Ω −→ R (Assigns a numerical value to each element of Ω)
For example, while tossing a coin: Sample space is Ω = {H, T }.
An appropriate random variable for this experiment is X(H) = 0 , X(T ) = 1.