Elementary Probability Theory

probability space

Warnings:

• probability theory is mean as hell

• it is extremely easy to fool yourself
• laying precise mathematical foundations took very long
• when is a sequence of bits ‚random‘?
• can we define this today?
• more in ‚theoretical computer science‘
 probability space

Warnings:

• probability theory is mean as hell

• it is extremely easy to fool yourself
• laying precise mathematical foundations took very long
• when is a sequence of bits ‚random‘?
• can we define this today?
• more in ‚theoretical computer science‘

the good news:

• today we know about ‚probability spaces‘

• in the simplest form topic of this lecture
• kind of a safety belt
• rule in my seminars in Saarbrucken
• if you say ‚probability‘
• and you cannot define the probability space you are using
• then your talk is over for the day and you can repeat it at the next session
 discrete probability spaces
def.

W =( S , p) probability space, describing a random experiment

S set, finite or countable, sample space

s∈S sample, possible outcome of the experiment

p : S →[ 0 , 1]
probability function

Σ s∈ S p( s)= 1 p(s): probability that the outcome of the experiment is s

 discrete probability spaces
def.

W =( S , p) probability space, describing a random experiment

S set, finite or countable, sample space

s∈S sample, possible outcome of the experiment

p : S →[ 0 , 1]
probability function

Σ s∈ S p( s)= 1 p(s): probability that the outcome of the experiment is s

example 1: coin flip (with unbiased coin)

S={0 , 1} p( 0)= p(1)=1/ 2

 discrete probability spaces
def.

W =( S , p) probability space, describing a random experiment

S set, finite or countable, sample space

s∈S sample, possible outcome of the experiment

p : S →[ 0 , 1]
probability function

Σ s∈ S p( s)= 1 p(s): probability that the outcome of the experiment is s

example 1: coin flip (with unbiased coin)

S={0 , 1} p( 0)= p(1)=1/ 2

example 2: throwing a dice (with a fair dice)

S={1 , 2 , 3 , 4 ,5 ,6 } ∀ s : p ( s)=1 / 6
 Events
def.

W =( S , p) probability space, describing a random experiment A⊆S event

S set, finite or countable, sample space p( A)=Σ a ∈ A p (s ) probability of A

s∈S sample, possible outcome of the experiment

a∈S elementary event

p : S →[ 0 , 1]
probability function

Σ s∈ S p( s)= 1 p(s): probability that the outcome of the experiment is s

example 1: coin flip (with unbiased coin)

S={0 , 1} p( 0)= p(1)=1/ 2

example 2: throwing a dice (with a fair dice)

S={1 , 2 , 3 , 4 ,5 ,6 } ∀ s : p ( s)=1 / 6
 Events
def.

W =( S , p) probability space, describing a random experiment A⊆S event

S set, finite or countable, sample space p( A)=Σ a ∈ A p (s ) probability of A

s∈S sample, possible outcome of the experiment

a∈S elementary event

p : S →[ 0 , 1]
probability function example for dice
Σ s∈ S p( s)= 1 p(s): probability that the outcome of the experiment is s
A ={1 ,3 ,5 } B ={2 , 4 , 6 }

example 1: coin flip (with unbiased coin)

p( A )= p( B )= 1/ 2

S={0 , 1} p( 0)= p(1)=1/ 2

example 2: throwing a dice (with a fair dice)

S={1 , 2 , 3 , 4 ,5 ,6 } ∀ s : p ( s)=1 / 6
 conditional probability
def.

W =( S , p) probability space, describing a random experiment A⊆S event

S set, finite or countable, sample space p( A)=Σ a ∈ A p (s ) probability of A

s∈S sample, possible outcome of the experiment

a∈S elementary event

p : S →[ 0 , 1]
probability function example for dice
Σ s∈ S p( s)= 1 p(s): probability that the outcome of the experiment is s
A ={1 ,3 ,5 } B ={2 , 4 , 6 }

example 1: coin flip (with unbiased coin)

p( A )= p( B )= 1/ 2

S={0 , 1} p( 0)= p(1)=1/ 2

A , B ⊆ S , p ( A )≠ 0
example 2: throwing a dice (with a fair dice)

p( B ∩ A)
S={1 , 2 , 3 , 4 ,5 ,6 } ∀ s : p ( s)=1 / 6 p( B∨ A)= probability of B given A
p( A )
 conditional probability
def.

W =( S , p) probability space, describing a random experiment A⊆S event

S set, finite or countable, sample space p( A)=Σ a ∈ A p (s ) probability of A

s∈S sample, possible outcome of the experiment

a∈S elementary event

p : S →[ 0 , 1]
probability function example for dice
Σ s∈ S p( s)= 1 p(s): probability that the outcome of the experiment is s
A ={1 ,3 ,5 } B ={2 , 4 , 6 }

example 1: coin flip (with unbiased coin)

p( A )= p( B )= 1/ 2

S={0 , 1} p( 0)= p(1)=1/ 2

A , B ⊆ S , p ( A )≠ 0
example 2: throwing a dice (with a fair dice)

p( B ∩ A)
S={1 , 2 , 3 , 4 ,5 ,6 } ∀ s : p ( s)=1 / 6 p( B∨ A)= probability of B given A
p( A )

1/ 6
p({1 }∨{1 ,3 ,5 })= =1/ 3
1/ 2
0
p({2 , 4 , 6 }∨{1 ,3 ,5 })= =0
1/ 2
 independent events

A , B ⊆S independent iff p( A ∩ B )= p ( A ) ⋅ p( B )
 independent events

A , B ⊆S independent iff p( A ∩ B )= p ( A ) ⋅ p( B )

single coin flip or single throw of dice:

too simple for example
 product of probability spaces

A , B ⊆S independent iff p( A ∩ B )= p ( A ) ⋅ p( B )

single coin flip or single throw of dice:

too simple for example

let’s consider two experiments

W 1 =( S 1 , p1) , W 2=( S 2 , p2 )
 product of probability spaces

A , B ⊆S independent iff p( A ∩ B )= p ( A ) ⋅ p( B )

single coin flip or single throw of dice:

too simple for example

let’s consider two experiments

W 1 =( S 1 , p1) , W 2=( S 2 , p2 )

set of outcomes for the pair of events

S= S1 × S2 ={(a , b)∨ a ∈ S1 , b ∈ S 2 }

p( a , b)= p 1( a) ⋅ p 2 (b)

W =W 1 × W 2=( S , p)
 product of probability spaces

A , B ⊆S independent iff p( A ∩ B )= p ( A ) ⋅ p( B ) examples

coin twice
single coin flip or single throw of dice:
too simple for example
S={0 , 0 ¿ ,( 0 , 1) ,(1 , 0) ,(1 , 1) }

let’s consider two experiments

p( a , b)= p 1( a) ⋅ p 2 (b)=1/ 2 ⋅ 1/ 2=1 / 4

W 1 =( S 1 , p1) , W 2=( S 2 , p2 )

set of outcomes for the pair of events

S= S1 × S2 ={(a , b)∨ a ∈ S1 , b ∈ S 2 }

p( a , b)= p 1( a) ⋅ p 2 (b)

W =W 1 × W 2=( S , p)
 product of probability spaces

A , B ⊆S independent iff p( A ∩ B )= p ( A ) ⋅ p( B ) examples

coin twice
single coin flip or single throw of dice:
too simple for example
S={(0 , 0),( 0 , 1) ,( 1, 0) ,(1 , 1)}

let’s consider two experiments

p( a , b)= p 1( a) ⋅ p 2 (b)=1/ 2 ⋅ 1/ 2=1 / 4

W 1 =( S 1 , p1) , W 2=( S 2 , p2 )
dice twice
set of outcomes for the pair of events
S={(1 , 1) , … ,(1 , 6) , … ,(6 ,1) , … ,(6 , 6)}

S= S1 × S2 ={(a , b)∨ a ∈ S1 , b ∈ S 2 }
p( a , b)= p 1( a) ⋅ p 2 (b)=1/ 6 ⋅ 1/ 6=36

p( a , b)= p 1( a) ⋅ p 2 (b)

W =W 1 × W 2=( S , p)
 product of probability spaces

A , B ⊆S independent iff p( A ∩ B )= p ( A ) ⋅ p( B ) examples

coin twice
single coin flip or single throw of dice:
too simple for example
S={(0 , 0),( 0 , 1) ,( 1, 0) ,(1 , 1)}

let’s consider two experiments

p( a , b)= p 1( a) ⋅ p 2 (b)=1/ 2 ⋅ 1/ 2=1 / 4

W 1 =( S 1 , p1) , W 2=( S 2 , p2 )
dice twice
set of outcomes for the pair of events
S={(1 , 1) , … ,(1 , 6) , … ,(6 ,1) , … ,(6 , 6)}

S= S1 × S2 ={(a , b)∨ a ∈ S1 , b ∈ S 2 }
p( a , b)= p 1( a) ⋅ p 2 (b)=1/ 6 ⋅ 1/ 6=1/ 36

p( a , b)= p 1( a) ⋅ p 2 (b)
coin and dice

W =W 1 × W 2=( S , p) S={(0 ,1) , … ,(0 ,6) ,( 1, 1) ,… ,(1 ,6 )}

p( a , b)= p 1( a) ⋅ p 2 (b)=1/ 2 ⋅ 1/ 6=1/ 12

 product of probability spaces

A , B ⊆S independent iff p( A ∩ B )= p ( A ) ⋅ p( B )

single coin flip or single throw of dice:

too simple for example

let’s consider two experiments

W 1 =( S 1 , p1) , W 2=( S 2 , p2 )

set of outcomes for the pair of events

S= S1 × S2 ={(a , b)∨ a ∈ S1 , b ∈ S 2 }

p( a , b)= p 1( a) ⋅ p 2 (b)

W =W 1 × W 2=( S , p)
 product of probability spaces

A , B ⊆S independent iff p( A ∩ B )= p ( A ) ⋅ p( B )

single coin flip or single throw of dice:

too simple for example

let’s consider two experiments

W 1 =( S 1 , p1) , W 2=( S 2 , p2 )

set of outcomes for the pair of events

S= S1 × S2 ={(a , b)∨ a ∈ S1 , b ∈ S 2 }

p( a , b)= p 1( a) ⋅ p 2 (b)

W =W 1 × W 2=( S , p)
 embedding events of single spaces in the product space

A , B ⊆S independent iff p( A ∩ B )= p ( A ) ⋅ p( B )
A ⊆ S1 , B ⊆ S 2 events of single experiments

single coin flip or single throw of dice:

e 1 ( A )= A × S 2 , e 2 ( B )= S 1 × B embedding into S
too simple for example

let’s consider two experiments

W 1 =( S 1 , p1) , W 2=( S 2 , p2 ) outcome of the other event does not matter

set of outcomes for the pair of events

S= S1 × S2 ={(a , b)∨ a ∈ S1 , b ∈ S 2 }

p( a , b)= p 1( a) ⋅ p 2 (b)

W =W 1 × W 2=( S , p)
 embedding events of single spaces in the product space

A , B ⊆S independent iff p( A ∩ B )= p ( A ) ⋅ p( B )
A ⊆ S1 , B ⊆ S 2 events of single experiments

single coin flip or single throw of dice:

e 1 ( A )= A × S 2 , e 2 ( B )= S 1 × B embedding into S
too simple for example

let’s consider two experiments

W 1 =( S 1 , p1) , W 2=( S 2 , p2 ) outcome of the other event does not matter

set of outcomes for the pair of events example: dice twice

S= S1 × S2 ={(a , b)∨ a ∈ S1 , b ∈ S 2 } A ={1} e n1 ( A)={(1 ,1) , …(1 , 6) }

p( a , b)= p 1( a) ⋅ p 2 (b)

W =W 1 × W 2=( S , p)
 embedding events of single spaces in the product space

A , B ⊆S independent iff p( A ∩ B )= p ( A ) ⋅ p( B )
A ⊆ S1 , B ⊆ S 2 events of single experiments

single coin flip or single throw of dice:

e 1 ( A )= A × S 2 , e 2 ( B )= S 1 × B embedding into S
too simple for example

let’s consider two experiments

W 1 =( S 1 , p1) , W 2=( S 2 , p2 )

set of outcomes for the pair of events

S= S1 × S2 ={(a , b)∨ a ∈ S1 , b ∈ S 2 }

p( a , b)= p 1( a) ⋅ p 2 (b)

W =W 1 × W 2=( S , p)
 embedding events of single spaces in the product space

A , B ⊆S independent iff p( A ∩ B )= p ( A ) ⋅ p( B )
A ⊆ S1 , B ⊆ S 2 events of single experiments

single coin flip or single throw of dice:

e 1 ( A )= A × S 2 , e 2 ( B )= S 1 × B embedding into S
too simple for example

let’s consider two experiments

W 1 =( S 1 , p1) , W 2=( S 2 , p2 )

set of outcomes for the pair of events

S= S1 × S2 ={(a , b)∨ a ∈ S1 , b ∈ S 2 }

p( a , b)= p 1( a) ⋅ p 2 (b) proof:

W =W 1 × W 2=( S , p)
 embedding events of single spaces in the product space

A , B ⊆S independent iff p( A ∩ B )= p ( A ) ⋅ p( B )
A ⊆ S1 , B ⊆ S 2 events of single experiments

single coin flip or single throw of dice:

e 1 ( A )= A × S 2 , e 2 ( B )= S 1 × B embedding into S
too simple for example

let’s consider two experiments

W 1 =( S 1 , p1) , W 2=( S 2 , p2 )

set of outcomes for the pair of events

proof:

S= S1 × S2 ={(a , b)∨ a ∈ S1 , b ∈ S 2 }

p( a , b)= p 1( a) ⋅ p 2 (b)

W =W 1 × W 2=( S , p)
 n independent experiments

W i =( S i , pi ) i ∈ {1 , … , n }
 n independent experiments

W i =( S i , pi ) i ∈ {1 , … , n }

W =W 1 × … × W n=( S , p)

S = S1 × … × S n p( a1 , … , an )= p 1 (a 1) ⋅ … , p n ( an )¿
 n independent experiments

W i =( S i , pi ) i ∈ {1 , … , n }

W =W 1 × … × W n=( S , p)

S = S1 × … × S n p( a1 , … , an )= p 1 (a 1) ⋅ … , p n ( an )¿
 n independent experiments

W i =( S i , pi ) i ∈ {1 , … , n }

W =W 1 × … × W n=( S , p)

S = S1 × … × S n p( a1 , … , an )= p 1 (a 1) ⋅ … , p n ( an )¿

embedding events for single experiments

Ai ⊆ Si i ∈ {1 , … , n }

e i ( A i )=S 1 × … Si−1 × Ai × S i+1 × … × S n ⊆ S

 n independent experiments

W i =( S i , pi ) i ∈ {1 , … , n }

W =W 1 × … × W n=( S , p)

S = S1 × … × S n p( a1 , … , an )= p 1 (a 1) ⋅ … , p n ( an )¿

embedding events for single experiments

Ai ⊆ Si i ∈ {1 , … , n }

e i ( A i )=S 1 × … Si−1 × Ai × S i+1 × … × S n ⊆ S

 n independent experiments

W i =( S i , pi ) i ∈ {1 , … , n } B1 , … , B s ⊆ S events

W =W 1 × … × W n=( S , p) are mutually independent iff

S = S1 × … × S n p( a1 , … , an )= p 1 (a 1) ⋅ … , p n ( an )¿ ∀ K ⊆ {1 , … , s } for all subsets of set of indices, resp. B’s

p( ⋂ B i)=Π i ∈ K p(Bi)
i∈ K
embedding events for single experiments

Ai ⊆ Si i ∈ {1 , … , n }

e i ( A i )=S 1 × … Si−1 × Ai × S i+1 × … × S n ⊆ S

 n independent experiments

W i =( S i , pi ) i ∈ {1 , … , n } B1 , … , B s ⊆ S events

W =W 1 × … × W n=( S , p) are mutually independent iff

S = S1 × … × S n p( a1 , … , an )= p 1 (a 1) ⋅ … , p n ( an )¿ ∀ K ⊆ {1 , … , s } for all subsets of set of indices, resp. B’s

p( ⋂ B i)=Π i ∈ K p(Bi)
i∈ K
embedding events for single experiments

Ai ⊆ Si i ∈ {1 , … , n }

e i ( A i )=S 1 × … Si−1 × Ai × S i+1 × … × S n ⊆ S

 random variables and their expected value

W =( S , p) probability space

random variable

the real numbers

simplest case:
• elementary events are numbers
• X is identity X (a)=a
 random variables and their expected value

W =( S , p) probability space

random variable

the real numbers

simplest case:
• elementary events are numbers
• X is identity X (a)=a
• or X is constant X (a)=c

expected value of random variable X

E ( X )= Σ a ∈ S X ( a)⋅ p( a)
 random variables and their expected value

W =( S , p) probability space examples

X (a)=c X is constant
random variable
E ( X )= Σ a ∈ S c ⋅ p( a)= c ⋅ Σ a∈ S p ( a)=c

the real numbers

simplest case:
• elementary events are numbers
• X is identity X (a)=a
• or X is constant X (a)=c

expected value of random variable X

E ( X )= Σ a ∈ S X ( a)⋅ p( a)
 random variables and their expected value

W =( S , p) probability space examples

X (a)=c X is constant
random variable
E ( X )= Σ a ∈ S c ⋅ p( a)= c ⋅ Σ a∈ S p ( a)=c

coin X (a)=a

the real numbers

E ( X )= Σ a ∈ {0 ,1 } a ⋅ 1 /2=0 ⋅ 1 /2+ 1⋅ 1/ 2=1/ 2
simplest case:
• elementary events are numbers
• X is identity X (a)=a
• or X is constant X (a)=c

expected value of random variable X

E ( X )= Σ a ∈ S X ( a)⋅ p( a)
 random variables and their expected value

W =( S , p) probability space examples

X (a)=c X is constant
random variable
E ( X )= Σ a ∈ S c ⋅ p( a)= c ⋅ Σ a∈ S p ( a)=c

coin X (a)=a

the real numbers

E ( X )= Σ a ∈ {0 ,1 } a ⋅ 1 /2=0 ⋅ 1 /2+ 1⋅ 1/ 2=1/ 2
simplest case:
• elementary events are numbers dice X (a)=a
• X is identity X (a)=a
• or X is constant X (a)=c
E ( X )= Σ a ∈ {1 ,… ,6 } a ⋅ 1/ 6=3 ⋅( 1+ 6)/ 6=7 / 2

expected value of random variable X

E ( X )= Σ a ∈ S X ( a)⋅ p( a)
 random variables and their expected value

W =( S , p) probability space examples

X (a)=c X is constant
random variable
E ( X )= Σ a ∈ S c ⋅ p( a)= c ⋅ Σ a∈ S p ( a)=c

coin X (a)=a

the real numbers

E ( X )= Σ a ∈ {0 ,1 } a ⋅ 1 /2=0 ⋅ 1 /2+ 1⋅ 1/ 2=1/ 2 ∉S

simplest case:
• elementary events are numbers dice X (a)=a
• X is identity X (a)=a
• or X is constant X (a)=c
∉S
E ( X )= Σ a ∈ {1 ,… ,6 } a ⋅ 1/ 6=3 ⋅( 1+ 6)/ 6=7 / 2

expected value of random variable X

E ( X )= Σ a ∈ S X ( a)⋅ p( a)
 linearity

W =( S , p) probability space

random variable

the real numbers

simplest case:
• elementary events are numbers
• X is identity
• or X is constant
expected value of random variable X
E ( X )= Σ a ∈ S X ( a)⋅ p( a)
 linearity

W =( S , p) probability space

random variable

the real numbers

simplest case:
• elementary events are numbers
• X is identity
• or X is constant
expected value of random variable X
E ( X )= Σ a ∈ S X ( a)⋅ p( a)
 linearity

W =( S , p) probability space

random variable

the real numbers

by induction:
simplest case:
• elementary events are numbers
• X is identity
• or X is constant
expected value of random variable X
E ( X )= Σ a ∈ S X ( a)⋅ p( a)
 random variables from different independent experiments

W =( S , p) probability space

random variable

expected value of random variable X

E ( X )= Σ a ∈ S X ( a)⋅ p( a)
 random variables from different independent experiments

W =( S , p) probability space

random variable

expected value of random variable X

E ( X )= Σ a ∈ S X ( a)⋅ p( a)
 random variables from different independent experiments

W =( S , p) probability space

random variable

expected value of random variable X

E ( X )= Σ a ∈ S X ( a)⋅ p( a)

sum if we perform independent experiments

 random variables from different independent experiments

W =( S , p) probability space

random variable

expected value of random variable X

E ( X )= Σ a ∈ S X ( a)⋅ p( a)

sum if we perform independent experiments

 random variables from different independent experiments

W =( S , p) probability space

random variable

expected value of random variable X

E ( X )= Σ a ∈ S X ( a)⋅ p( a)

sum if we perform independent experiments