Basic Concepts of Probability

Theory
 In this chapter
• Specifying Random experiments
• The axioms of probability
• Computing probabilities using counting
methods
• Conditional probability
• Independence of events
 Specifying Random Experiments
• Random experiment = Procedure +
Observation(s) or measurement(s)
• Two Random Experiments
– procedure may be the same
– different observations
• Example:
– Toss a coin 3 times and note sequence of Heads
and tails.
– Toss a coin 3 times and note number of Heads.
 Sample Space
• Outcome (or Sample Point): result that cannot be
decomposed into other results.
• Outcomes cannot occur simultaneously: mutually
exclusive.
• Sample Space (S): set of all possible outcomes.
– List of elements, A = {0, 1, 2, 3}
– Property verified by an element,
A = {x: x  N, 0 ≤ x ≤ 3}
• S can be discrete (countable)/continuous,
finite/infinite, multi-dimensional.
 Events
• Event: outcome satisfies certain conditions.
• Need to define subsets of S.
• Two special events:
– Certain Event, S, all outcomes.
– Impossible (or Null) Event, no outcomes.
• Elementary Event: single outcome in a
discrete Sample Space.
 Review of Set Theory
• Universal Set (U): Set of all possible objects.
• Set (A): collection of objects of U.
• Elements (or Points) of A: x  A.
• A Subset of B: A  B if every elements of A is
element of B.
• Empty set (): contains no element.  is a
subset of any set.
• A = B if and only if A  B and B  A.
 Review of Set Theory
• Union of two sets:
A  B = {x: x  A or x  B}.
• Intersection of two sets:
A  B = {x: x  A and x  B}.
• Complement of a set: Ac = {x: x  A}.
• Difference (or relative complement) of two
set: A - B = {x: x  A and x  B}.
 Basic Set Operations
• Commutative properties:
AB=BA
AB=BA
• Associative properties:
A  (B  C) = (A  B)  C
A  (B  C) = (A  B)  C
• Distributive properties
A  (B  C) = (A  B)  (A  C)
A  (B  C) = (A  B)  (A  C)
 DeMorgan’s Rules
(A  B)c = Ac  Bc
(A  B)c = Ac  Bc
Proof of the 1st rule:
If x  (A  B)c then x  A  B
x  A  x  Ac
x  B  x  Bc
 x  Ac  B c
 Event classes
• Class F of events:
– Only events in F are assigned probabilities.
– Any set operation on events in F will produce
events in F.
• In other words: F is a set of sets.
 Event classes: Example
• Coin toss.
• Sample set: S = {T, H}
• Event class: S ={, {H}, {T}, S}
 The axioms of probability
• Let
– E a random experiment,
– S: sample space,
– F: event class.
• Probability law: for each A  F, P[A] such as:
– 0 ≤ P[A]
– P[S] = 1
– If A1, A2, …

sequence
 
of events verifying (Ai  Aj)ij,
then P A k    P  A k 
 k 1  k 1
 Corollaries
• P[Ac] = 1 – P[A]
• P[A] ≤ 1
• P[] = 0
• If A1, A2, …, An (n  2) pairwise mutually
  n n

exclusive then     P  A 

P A
k 1
k
k 1
k

• P[A  B] = P[A] + P[B] - P[A  B]

 n
 n

•     P A    P A A   ...   1 P A ...A 

n 1
P A k j j k 1 n
k 1 j 1 j k

• If A  B then P[A] ≤ P[B].

 Discrete Sample Spaces
• Finite Sample Space : S ={a1, a2, …, an}
• B = {a1, a2, …, am} 
P[B] = P[{a1}]+P[{a2}]+…+P[{am}]
• Equally likely outcomes:
P[{a1}]=P[{a2}]=…=P[{an}]=1/n
• P(B) = m/n
• Example.
 Example
• An urn contains 10 identical balls labeled 0,1,
…, 9
• Select a ball. Consider the following events:
– A = “number of ball selected is odd”
– B = “number of ball selected is a multiple of 3”
– C = “number of ball selected is less than 5”
Find the probability of the events. And of
–AB
–ABC
 Example
• P[A] = 5 / 10
• P[B] = 3 / 10
• P[C] = 5 / 10
• P[A  B] = 6 / 10
• P[A  B  C] = 9 / 10
 Continuous Sample Spaces
• Experiment: pick a number at random
between 0 and 1.
• S = [0, 1].
• Probability law: The probability that the
outcome falls in [a, b] is equal to b – a.
• Axiom I: P[[a, b]]  0,
• Axiom II: P[S] = P[[0, 1]] = 1,
• Axiom III: P[[0, 0.2]  [0.8, 1]] = 0.2 + 0.2 = 0.4
 Computing Probabilities using
Counting Methods
• Sampling with replacement and with ordering
• Sampling without replacement and with
ordering
• Permutations of distinct objects
• Sampling without replacement and without
ordering
 Sampling with replacement and
with ordering
• Choosing k objects from a set of n distinct objects
with replacement  ordered list.
• The Population: A = {x1, x2, …, xn}
• Ordered k-tuple (x1, x2, …, xk)
• Number of ordered k-tuple = nk.
• Example: Select 2 balls from an urn that contains
5 labeled balls. Number of outcomes?
• Answer: 25
 Sampling without replacement and
with ordering
• Choosing k objects from a set of n distinct
objects without replacement  ordered list.
– 1st draw: n possible outcomes,
– 2nd draw: (n – 1) possible outcomes,
– kth draw: (n – k + 1) possible outcomes.
• Number of distinct ordered k-tuples:
n (n - 1)…(n – k + 1)
 Sampling without replacement and
with ordering
• Example: Select 2 balls without replacement
from an urn that contains 5 labeled balls.
Number of outcomes?
• Answer: 5(4) = 20.
 Permutations of n distinct objects
• Sampling without replacement with k = n.
number of permutation = n(n-1)…(2)(1) = n!
• Stirling’s formula:
n !  2 n n 1 2e  n
• Number of permutations of 4 distinct objects
{1, 2, 3, 4}: 4! = 4 × 3 × 2 × 1 = 24.
 Sampling without replacement and
without ordering
• Select k objects out of a set of n distinct
objects without replacement without regard
to order  combination of size k (k < n).
n  n!
C  
n

 k  k ! n  k  !
k

n 
•   (n choose k): binomial coefficient.
k 
 Sampling with replacement and
without ordering
• The number of ways of choosing k objects out
of a set of n distinct objects with replacement
and without ordering:
 n 1 k 
 
 k 
 Conditional Probability
• The conditional probability of event A given
that event B has occurred:
P  A B 
PA |B  ,P  B   0
PB

• Example: An urn contains 2 black balls (1 and

2) and 2 white balls (3, 4).
– A = {(1, b), (2, b)}
– B = {(2, b), (4, w)}
– C = {(3, w), (4, w)}
– Find P[A|B] and P[A|C].
 Conditional Probability
• Note that: P[A  B] = P[A|B]P[B]
P[A  B] = P[B|A]P[A]
• Example: An urn contains 2 black balls and 3 white balls. Pick
2 balls without replacement and note the sequence of colors.
Q: Probability that both balls are black?
SOL.:
P(B1  B2 )=P(B2 l B1 ) P(B1)
P(B1)= 2/5
P(B2 l B1 ) = ¼
P(B1  B2 ) =2/5 X ¼ = 2/20 = 1/10
 Tree Diagram

0
B1 2/5 3/5 W1

1 2
B2 W2 B2 W2
1/4 3/4 2/4 2/4

1/10 3/10 3/10 3/10

 Conditional Probability
• B1, B2, … Bn a partition of the Sample Set S if:
– Are pairwise mutually exclusive,
– S = B1  B2  … B n
• For any event A:
A = A  S = A  (B1  B2  … Bn)
A = (A  B1)  (A  B2)  …  (A  Bn)
P[A] = P[A  B1] + P[A  B2] + … + P[A  Bn]
P[A] = P[A|B1]P[B1] + … + P[A|Bn]P[Bn]
 Example
• An urn contains 2 black balls and 3 white balls.
Pick 2 balls without replacement and note the
sequence of colors.
• Q: Probability of W2 that the 2nd ball is white.
• Solution: B1={(b,b), (b,w)} and W1={(w,b),
(w,w)} form a partition of S.
P[W2] = P[W2|B1]P[B1] + P[W2|W1]P[W1]
P[W2] =(3/4)(2/5)+(1/2)(3/5)=3/5
 Independence of events
• Events A and B are independent if
P[A  B] = P[A] P[B]
• Remember that P[A  B] = P[A|B]P[B], thus
P[A|B] = P[A] and P[B|A] = P[B]
 Example 1
• Example: An urn contains 2 black balls (1 and
2) and 2 white balls (3, 4).
– A = {(1, b), (2, b)}
– B = {(2, b), (4, w)}
– C = {(3, w), (4, w)}
• Q: A and B independent? A and C ?
 Example: Solution
• P[A] = P[B] = 0.5
• P[A  B] = P[{(2, b)}] = 0.25
• P[A  B] = P[A] P[B] = (0.5) (0.5) = 0.25
A and B are independent
• Verify that P[A|B] = P[A]
• Verify that A and C are not independent.
• Generalization: If 2 events have nonzero
probability and are mutually exclusive then they
are not independent.
 Example 2
• Two numbers x and y selected between 0 and
1.
– A = {x > 0.5}
– B = {y > 0.5}
– C = {x > y}
• Q: A and B independent? A and C?
• Solution: A and B independent,
A and C are not independent
 Independence of three events
• A, B and C are independent if
1. P(A  B) = P(A) P(B)
2. P(A  C) = P(A) P(C)
3. P(B  C) = P(B) P(C)
4. P(A  B  C) = P(A) P(B) P(C)
 Example
• Two numbers x and y selected between 0 and
1.
– B = {y > 0.5}
– D = {x < 0.5}
– F = {x < ½ and y < ½}  {x > ½ and y > ½}
• B, D and F are pairwise independent, but the
three events are not independent.
 Sequential experiments
• Sequence of simpler experiments (independent
or dependent)
• Sequence of independent experiments:
E1, E2, …, En.
• The outcome s = (s1, s2, …, sn)
• Sample space: Cartesian product of sample
spaces: S1 × S2 × … × Sn.
• P(A1  A2  …  An) = P(A1) P(A2)… P(An)
 Sequence of independent
experiments: Example
• 10 numbers selected at random from [0, 1]
• Find the probability that the first 5 numbers are
less than ¼ and the last 5 are greater than ½.
• For k = 1, 2, …, 5 : Ak = {xk < ¼)
• For k = 6, 7, …, 10 : Ak = {xk > ½)
• P(A1  A2  …  A10) = P(A1) P(A2)… P(A10)
• P(A1  A2  …  A10) = (1/4)5(1/2)5
 Bernoulli Law
• Bernoulli trial: performing an experiment once
and noting whether an event A occurs.
• If A occurs: “success”
• Else: “failure”.
 Bernoulli Law: Example
• Toss of a coin, A = {H}, P(A) = p.
• Coin tossed 3 times.
– P[{HHH}] = p3.
– P[{HHT}] = p2(1-p).
– Etc.
 Bernoulli Law: Example
• k: number of heads.
• P[k = 0] = (1 – p)3
• P[k = 1] = 3p(1 – p)2
• P[k = 2] = 3p2(1 – p)
• P[k = 3] = p3
 Bernoulli Law: Theorem
• k successes in n independent Bernoulli trials:
n  k
p n  k     p (1  p ) n k for k  0,1,..., n
k 
Where:
n  n!
 
 k  k ! n  k  !
 Example
• 8 independent speakers.
• A speaker is active with a probability of 1/3.
• What is the probability that more than 6
speakers are active?

P  P7  P8
7 8
 8  1  2 8 1 
P        
 7  3  3 8 3 
 Multinomial Probability Law
• B1, …, BM: partition of Sample Space.
• For j between 1 and M
– P[Bj] = pj
– kj number of times an event Bj occurs
n!
P  k 1 , k 2 ,..., k M    p1k 1 p 2k 2 ... p Mk M
k 1 !k 2 !....k M !
 Geometrical Probability Law
• Sequence of Bernoulli experiments
• Repeat trials until the first success.
• The outcome m: number of trials till the first
success.
p  m   P  A1 A 2 ...A m 1A m    1  p 
C C C m 1
p
 
1
 p  m   p q
m 1 m 1
m 1
p
1 q
1
 
1
P  m  K    p 
m  K 1
q m 1  pq K  q j  pq K
j 0 1 q
qK
 Problem-1

*- In how many ways can a team of eleven

be picked from sixteen possible players?
 Problem 2

*- Determine the probabilities of selecting at

random (a) a man, and (b) a woman from a
crowd containing 20 men and 33 women.
 Problem 3
*- A batch of 40 components contains 5 that are
defective. If a component is drawn at random
from the batch and tested and then a second
component is drawn at random, Calculate the
probability of having one defective
component, both with and without
replacement.