Notes 2

Introduction to Probability
Aria Nosratinia Probability and Statistics
2-1
Modeling in Science and Engineering

We can only know the world through observations Scientic laws are patterns we observe via measurements Laws of Electromagnetics Newtons laws of motion / special relativity Law of gravity / general relativity Law of conservation of matter/energy and momentum These laws are models of reality that we use to predict its behavior, or to build machines that work in the physical world. Thus, physical laws are tools for understanding the world. Probability is also a tool It is a model that we construct to understand the world
Aria Nosratinia Probability and Statistics 2-2
Why Probability
Q: Why arent deterministic models enough? A: Sometimes we cannot know the truth exactly. Either it is unknowable Or it is too expensive to model (kinetic theory of gases) But we still want to make intelligent decisions or predictions Probability has been designed to help us in these situations
2-3
Examples of Usage of Probability
Will it rain tomorrow?

Who will win the next US presidential election?
2-5
How many defective chips from a wafer?
2-6
History
Probability was motivated by games of chance (dice) Started by correspondence between Pierre de Fermat and Blaise Pascal (1654) Laplace wanted to know how to bet on various combinations of two dice. Basic idea: counting equally probable events If there are n equally probable outcomes, each has probability This served as an early denition of probability
1 n.
2-7
A fair coin toss has two equally likely outcomes, each with probability 1 2
A fair die throw has 6 equally likely outcomes, each with probability 1 6.
2-8
Relative Frequency
Sometimes the events of interest are not equally likely. Example: Probability of at least one heads in two coin tosses. Using equally likely events, calculate relative frequency Probability = Number of desirable events Total number of events
In the example, above, all events are {T T, T H, HT, HH } and desirable events are {T H, HT, HH }. Thus it has probability 3/4
2-9
A storm was brewing...
This seemed ne, development of probability continued Combinatoric analysis was started But cracks in the facade started to appear. Certain inconsistencies were observed, some apparently easy results were too hard to show. It became clear the one needs a better foundation
2-10
Calmer waters
Setting: late 19th / early 20th century Set theory led to a sound foundation for Theory of numbers (real numbers) Integration Probability Major players: Georg Cantor, Emile Borel, Henri Lebesgue, Andrei Kolmogorov, ... We will now proceed to describe modern probability theory
2-11
Random Experiments
We use the die throw as an example. The actual throw of a die is a random experiment. Its outcome can have one out of six values. Outcomes: 1, 2, 3, 4, 5, 6 Any combination of outcomes is an event. We are interested in events, like: Did we get a 2? Did we get an even number? (2 or 4 or 6) Did we get a number less than 3? (1 or 2)
2-12
Outcomes and Events
Random Experiment
Event A={less than 4}
2 6 1
3 5
2-13
Events for Single Die Throw

Events include all our cases of interest for probability Example: Events for the die throw are: Size-0 event Size-1 events Size-2 events Size-3 events . . . Size-6 event {1} {1, 2} {1, 2, 3} {2} {1, 3} {1, 2, 4} {6} {5, 6} {4, 5, 6}
{1, 2, 3, 4, 5, 6}
In this case we have 64 events, because: 6 6 6 + + + 6 1 0

= 26
2-14
Review of Set Theory

Sets are collections of objects Sets are not ordered If a is a member of set A we write a A. We say B is a subset of A, and write B A, if: bB bA The number of members in A is called the magnitude of A and is shown with |A|. The set of all subsets of A is called the power set of A and has 2|A| members. The biggest possible set is called the universal set. When studying probability, the universal set is .
Set Operations
We dene union and intersection thus: AB = {x : x A and x B } AB = {x : x A or x B }
Set complement requires a universal set: A c = {x : x , x / A} We can also dene a set subtraction: A B = A Bc De Morgans laws: (A B ) c = A c B c (A B ) c = A c B c
2-16
Venn Diagram
A A B U B
AUB
2-17
Sets and Their Probabilities

To repeat: each event corresponds to a set. Example: in a die throw: {2} {less than 5}, {1, 5, 6}
To each valid event, we assign a number, its probability. So far, we have three ingredients for our probability soup: A space of all outcomes, denoted Events, each of them a subset of To each event A, we assign a probability, often shown P (A). We will now study the properties of probability.
Probability Axioms
Non-negativity: P (A) 0. Scaling: P () = 1 Countable Additivity: If Ai are mutually exclusive, i.e., if Ai Aj = for all i, j , then P (A 1 A 2 ) = P (A 1 ) + P (A 2 ) +
2-19
Math-speak
Denition: tells us what a certain term or sign means. Axiom: a fact that cannot be proven, but is taken to be true as a foundation for a theory (a logical necessity) Theorem: A statement of fact(s) that can be logically deduced from previously proven facts. Lemma: is a smaller result to be used in the proof of theorems. Corollary: is a result that easily follows from a theorem. Conjecture: is a result that experts feel there is sucient reason to believe is true, but has not been proven yet. Fermats last theorem was a conjecture until 1994, when Andrew Wiles nalized its proof.
Sample Space and Event Space

Sample space: another name for the space of all outcomes . Event space all possible events of interest in an experiment. Any set of non-overlapping events that cover is called a partition of . NOTE: The Yates and Goodman textbook uses the name event space for a partition. We try to avoid this mixup. Example: For a die throw, the event space is: , {1}, {2}, . . . , {6}, {1, 2}, {1, 3} . . . , {5, 6}, . . . , {1, 2, 3, 4, 5, 6} One possible partition is: {1, 2}, {3, 4}, {5, 6}
2-21
Consequences of Probability Axioms

Theorem: The probability measure P satises: P () = 0
P (A c ) = 1 P (A )
For any A and B , P (A B ) = P (A ) + P (B ) P (A B )
If A B then P (A) P (B )
2-22
Further Results
The probability of an event B = {s1 , . . . , sm } is the sum of the probabilities of all outcomes in that event.
If a sample space has n equally likely outcomes, the probability of 1 . each is n
2-23
Partitioned Probabilities
Theorem If {B1 , B2 , . . . , Bn } is a partition of and A is an arbitrary event, then: P (A ) = P (A B 1 ) + P (A B 2 ) + + P (A B n )
Proof: Events Ci = A Bi are mutually exclusive (why?) Therefore we can use axiom 3 to get the result.
2-24
2-25
B1 B2 B3 B4 B5
2-26
A
C1 C2 C3 C4 C5
2-27
Conditional Probability
Denition: The conditional probability of event A given B is: P (A |B ) = P (A B ) P (B )
NOTE: Often we write P (A, B ) instead of P (A B ). Interpretation: If we know event B has occurred, how does that change our knowledge of the probability of A.
2-28
Example (1)
Throwing a fair die A = { less than 4 } B = { odd number }
Clearly P (A) = 1/2 and P (B ) = 1/2, and P (A, B ) = 1/3, then: P (A |B ) = 2 1/3 = 1/2 3
If we know the number was odd, there is 2/3 chance it was less than 4. Note: The probability of A is increased if we know an odd number has occurred.
2-29
Example (2)
Throwing a fair die A = { less than 4 } B c = { even number }
Clearly P (A) = 1/2 and P (B c ) = 1/2, and P (A, B c ) = 1/6, then: P (A |B c ) = 1/6 1 = 1/2 3
If we know the number was even, with prob. 1/3 it was less than 4. Note: The probability of A is reduced if we know an even number has occurred.
2-30
Example (3)
Drawing a card at random from a deck of cards A = { red card } B = { Ace or king }
Then P (A, B ) = 4/52 and P (B ) = 8/52 P (A |B ) = 4/52 = 1/2 8/52
Note: We see that P (A) = P (A|B ), so the probability of A is unaected by a knowledge of B . This is an important case to be revisited later.
2-31
Properties of Conditional Probability
Theorem: Conditional probability has the following properties. P (A |B ) 0 P (B |B ) = 1 If A = A1 A2 and Ai Aj = for all i, j , P (A |B ) = P (A 1 |B ) + P (A 2 |B ) +
2-32
Law of Total Probability
Theorem: For a partition = B1 Bm , with P (Bi ) > 0 for all i,

m
P (A ) =
i=1
P (A |B i ) P (B i )
Proof: Use partition theorem with P (A|Bi )P (Bi ) = P (A, Bi ).
Application: whenever want to calculate the total probability from conditionals. Often used in mixtures.
2-33
Example
Whizywhiz Corp. has 3 factories producing semiconductor chips. The outputs are mixed, packaged, and shipped to customers. Factory A B C Chips/day 10,000 15,000 8,000 Defect probability 0.02 0.03 0.01
You buy one chip. What is the probability P (D) that it is defective? First calculate the probability of where it came from. 10,000 P (A) = 10,000+15 ,000+8,000 = 0.303 P (B ) = ? P (C ) = ? Then calculate the probability of failure. P (D ) = ?
Bayes Theorem
Theorem: P (B |A ) =
P (A |B )P (B ) P (A )
Proof: P (B |A ) =
P (A |B )P (B ) P (B, A) = P (A ) P (A )
Application: when we know P (A|B ), but want P (B |A).
2-35
Bayes Example
We go back to the example of WhizyWhiz chips. Lets say that you test one of the chips, and it comes up defective. What is the probability that it came from Factory A? 10, 000 = 0.303 10, 000 + 15, 000 + 8, 000 P (D|A) = 0.02 from table P (A ) = P (D ) = ? P (A |D ) = ?
Bayes theorem makes a seemingly hard problem very easy!

Variation on Bayes Rule
P (Bi |A) =
P (A|Bi )P (Bi ) n j =1 P (A|Bj )P (Bj )

P (A )
2-37
Bayes Example (2)

Background information: A plane is lost, and assumed crashed, in one of three wooded regions i = 1, 2, 3 with probabilities 0.2, 0.3, 0.5. The probability that a search will miss a crashed plane due to natural obstacles is 0.1, 0.2, 0.1 in the three regions. Question: We search region 1 and the search was unsuccessful. Now what is the probability that the plane is in region 1, 2, or 3?
2-38
Bayes Example (2)

Background information: A plane is lost, and assumed crashed, in one of three wooded regions i = 1, 2, 3 with probabilities 0.2, 0.3, 0.5. The probability that a search will miss a crashed plane due to natural obstacles is 0.1, 0.2, 0.1 in the three regions. Question: We search region 1 and the search was unsuccessful. Now what is the probability that the plane is in region 1, 2, or 3? Solution: The probability P (E1 ) of unsuccessful search in region 1: P (E1 ) =? Then nd probability of being in regions 1,2,3: P (R1 |E1 ) =? P (R2 |E1 ) =? P (R3 |E1 ) =?
Independence
Events A and B are independent when P (A B ) = P (A ) P (B ) This means, P (B |A ) = P (B ) P (A |B ) = P (A )
Assuming P (A) = 0 and P (B ) = 0, respectively. A zero-probability event is independent of everything. IMPORTANT NOTE: independent and disjoint are not the same! Disjoint events are often NOT independent (why?)
2-39
Example
We pick one card at random from a deck of cards. A = {The card is an ace} B = {The card is a spade} Are these two events independent?
2-40
Example
This is a more challenging example... We draw ve cards at random from a deck of cards A = {The hand is a ush} B = {The hand is a straight} Are the two probabilities independent? (This problem will wait until we have learned counting methods.)
2-41
Example
We have one fair coin, and a second coin that comes up tails with probability 0.7. We pick up one coin from our pocket and ip it, getting tails then put it back, and repeat, again getting tails. Are these two events independent?
2-42
Exercise
Consider a throw of a fair die = {1, 2, 3, 4, 5, 6} And consider the event A = {1, 2, 3, 4} Find at least one non-empty event that is independent of A. Can you nd another?
2-43
Independence of Multiple Events
Denition: Three events A, B, C are independent if all of the following are satised: P (ABC ) = P (A) P (B ) P (C ) P (AB ) = P (A) P (B ) P (AC ) = P (A) P (C ) P (BC ) = P (B ) P (C )
Why do you think P (ABC ) = P (A) P (B ) P (C ) is not enough?
2-44
Example (1)
Consider the space of equally probable outcomes = {1, 2, 3, 4, 5, 6, 7, 8} and the three events A = {1, 2, 3, 4}, B = {1, 2, 3, 4}, C = . We have P (A B C ) = P (A )P (B )P (C ) = 0 But are the events independent? (Note A = B ). To verify: P (A B ) = P (A ) = 0. 5 P (A)P (B ) = 0.25 The events are not independent.
2-45
Example (2)
Now consider the same space = {1, 2, 3, 4, 5, 6, 7, 8} with A = {1, 2, 3, 4}, B = {1, 2, 5, 6}, C = {1, 3, 5, 7}. 1 2 1 P (AB ) = P (AC ) = P (BC ) = 4 1 P (ABC ) = 8 P (A ) = P (B ) = P (C ) = Verify that these events are independent.
2-46
Independence of m Events
Events A1 , A2 , . . . , Am are independent if: P (A i A j ) = P (A i ) P (A j ) P (A i A j A k ) = P (A i ) P (A j ) P (A k ) P (A 1 A 2 A m ) = P (A 1 ) P (A 2 ) P (A m ) i, j i, j, k
2-47
Conditional Independence
Denition: Events A and B are independent conditioned on C if: P (A B |C ) = P (A |C ) P (B |C )
In general, it is not possible to infer conditional independence from unconditional independence, or vice versa.
We shall see how that may happen, via examples
2-48
Conditional Independence Example 1
Consider the throw of a fair die: = {1, 2, 3, 4, 5, 6} Example 1: Consider the following events: A = {1, 2, 3, 4} Verify the following P (AB ) = P (A)P (B ) P (AB |C ) = P (A|C )P (B |C ) B = {1, 2, 5} C = {1, 2, 3, 4, 5}
2-49
Conditional Independence Example 2

Consider once again the throw of a fair die: = {1, 2, 3, 4, 5, 6} Example 2: Consider the following events: A = {1, 2} Verify that: P (AB ) = P (A)P (B ) P (AB |C ) = P (A|C )P (B |C ) Conditioning sometimes can create independence, and sometimes can destroy independence. B = {2, 3} C = {1, 2, 3, 4}
2-50
Tree Diagrams
Tree diagrams can be used for showing interconnected events. Example: Two coins, one fair, one 70% tails. We pick one at random and ip it. What is the probability of outcomes.
C1 0.5
0.7 0.3 0.5
C1T C1H C2T C2H
0.5 C2
0.5
2-51
Counting
Fundamental principle: If one experiment has k outcomes, and another has n outcomes, then a combined experiment has nk outcomes
Example 1: The throw of one die has 6 outcomes. Throwing two dice has 36 outcomes. Example 2: Throw of a die has 6 outcomes, ip of a coin has two. A throw and a ip together have 12 outcomes.
2-52
Sampling
Taking one out of several objects is sometimes called sampling. Sometimes we might do sampling with replacement EXAMPLE: draw one card at random. return, shue, and draw another card at random. Sometimes we may do sampling without replacement EXAMPLE: draw two cards successively at random If the order of outcomes is important (e.g., in a two-card draw, King-Ace vs. Ace-King) we call it permutations. If the order of outcomes is unimportant, we call it combinations.
Permutations
Example: drawing three cards from a deck of cards, and putting them down in order, has how many outcomes? 52 51 50 = 132, 600 NOTE: In this case, we count 6 sequences per each combination, because now order is important, i.e., for three cards A, B, C , ABC ACB BAC BCA CAB CBA
Theorem: The number of k -permutations of n distinguishable objects is n! (n)k = n(n 1) (n k + 1) = (n k )!
2-54
Combination
Example: drawing three cards from a deck of cards, and putting them down disregarding order, has how many outcomes? 52 51 50 = 22, 100 321 Reasoning: For each 3-card combination, there are 3 ways to choose the rst card, and 2 ways to choose the second, and one way to choose the last one. Theorem: The number of ways to choose k objects out of n distinguishable objects is: n k = n! k !(n k )!
This expression is called n choose k.

Example (Poker)
We draw ve cards at random from a deck of cards A = {The hand is a ush} B = {The hand is a straight} What are the two probabilities? Are the two events independent?
2-56
Example (Poker)
Number of poker hands = Number of ush hands =
52! 5!((525)!
= 2, 598, 960 = 5, 148
521211109 5!
Number of straights (ace high or low) = 40 4 4 4 4 = 10, 240 Number of straight ushes (ace high or low) = 10 4 = 40 Prob(ush) =
5,148 2,598,960
= 0.198% = 0.394% = 0.0015%
Prob(straight) =
10,240 2,598,960
prob (straight ush) =
40 2,598,960
We see that the two events are not independent.
2-57
Sampling With Replacement
Theorem: There are mn ways to choose, with replacement, an ordered sample of length-n out of m distinguishable objects. Example: How many binary sequences of length 8? (256) Example: How many distinct four-letter words from English alphabet? (264 = 456, 976) Example: If a sample space has n outcomes, repeating the experiment k times will have n n n = nk outcomes.
2-58
Sampling with Replacement (2)

We may be interested in specic outcomes in sampling with replacement E.g. How many binary sequence of length-8 that have exactly ve 1 and three 0. In this case, if a number is not zero, it is one, so there are binary numbers with ve 1 and three 0. This is the number of ways to choose 3 slots out of 8.
n k 8 3
is known as the binomial coecient
2-59
Sampling with Replacement (3)

Now, let us consider experiments with m > 2 outcomes. In general, consider = {s 1 , . . . , s m } We repeat this experiment n times. For n = 6 and m = 4, this is an example outcome: (s 4 , s 3 , s 1 , s 1 , s 2 , s 4 ) We want to know how many outcomes are there in which s1 is repeated n1 times, s2 is repeated n2 times, .... Naturally n1 + n2 + . . . = n. The answer is the multinomial coecient n n0 , . . . , n m = n! n0 ! n 1 ! n m !
2-60
Independent Trials
We now apply combinatorics in nding certain probabilities with independent trials QUESTION: What is the probability of n0 heads and n1 tails in n0 + n1 tosses of a coin that comes tails with prob. p? Probability of any xed sequence of n0 heads and n1 tails is: (1 p)n0 pn1 Probability of all such sequences is: n (1 p)n0 pn1 n0
2-61
Independent Trials (2)

Theorem: The probability of n0 failures and n1 successes in n = n0 + n1 trials with success probability p is n (1 p)nn1 pn1 = n1 n (1 p)n0 pnn0 n0
Exercise: Whizywhiz chips have a failure probability of 0.02. Out of a batch of 100 chips, what are the chances that 95 of them are good? What are the chances that at least 95 of them are good?
2-62
Independent Trials (3)

Theorem: Consider a sample space = {s1 , . . . , sm } with probabilities P (si ) = pi . For n = n1 + + nm independent trials, the probability of si occurring exactly ni times is: n m p n1 p n m n1 , . . . , n m 1
Exercise: What is the probability that in 12 throws of a fair die, each face will appear exactly twice?
2-63
A Puzzle
We nish this chapter with an interesting puzzle. Three prisoners are told by the prison guard that one of them has been chosen at random and will be freed, while the other two are convicted. Prisoner A requests to know privately the name of one of his fellow prisoners who is convicted, claiming it does not aect his fate because he already knows at least one of the others are convicted. But the guard refuses, saying that the answer would change the probability of the prisoners freedom from 1/3 to 1/2. Which one of them is correct, and why?
2-64

Notes 2

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Notes 2

Uploaded by

Copyright:

Available Formats

Introduction to Probability

Aria Nosratinia Probability and Statistics

Modeling in Science and Engineering

Aria Nosratinia Probability and Statistics

Examples of Usage of Probability

Will it rain tomorrow?

Examples of Usage of Probability

Who will win the next US presidential election?

Aria Nosratinia Probability and Statistics

Examples of Usage of Probability

How many defective chips from a wafer?

Aria Nosratinia Probability and Statistics

Aria Nosratinia Probability and Statistics

Aria Nosratinia Probability and Statistics

Aria Nosratinia Probability and Statistics

A storm was brewing...

Aria Nosratinia Probability and Statistics

Aria Nosratinia Probability and Statistics

Aria Nosratinia Probability and Statistics

Outcomes and Events

Event A={less than 4}

Aria Nosratinia Probability and Statistics

Events for Single Die Throw

In this case we have 64 events, because: 6 6 6 + + + 6 1 0

Review of Set Theory

Aria Nosratinia Probability and Statistics

Aria Nosratinia Probability and Statistics

Sets and Their Probabilities

Aria Nosratinia Probability and Statistics

Sample Space and Event Space

Aria Nosratinia Probability and Statistics

Consequences of Probability Axioms

For any A and B , P (A B ) = P (A ) + P (B ) P (A B )

Aria Nosratinia Probability and Statistics

If a sample space has n equally likely outcomes, the probability of 1 . each is n

Aria Nosratinia Probability and Statistics

Theorem If {B1 , B2 , . . . , Bn } is a partition of and A is an arbitrary event, then: P (A ) = P (A B 1 ) + P (A B 2 ) + + P (A B n )

Aria Nosratinia Probability and Statistics

Aria Nosratinia Probability and Statistics

Aria Nosratinia Probability and Statistics

Aria Nosratinia Probability and Statistics

Denition: The conditional probability of event A given B is: P (A |B ) = P (A B ) P (B )

Aria Nosratinia Probability and Statistics

Aria Nosratinia Probability and Statistics

Aria Nosratinia Probability and Statistics

Then P (A, B ) = 4/52 and P (B ) = 8/52 P (A |B ) = 4/52 = 1/2 8/52

Aria Nosratinia Probability and Statistics

Properties of Conditional Probability

Aria Nosratinia Probability and Statistics

Law of Total Probability

Theorem: For a partition = B1 Bm , with P (Bi ) > 0 for all i,

Proof: Use partition theorem with P (A|Bi )P (Bi ) = P (A, Bi ).

Aria Nosratinia Probability and Statistics

Application: when we know P (A|B ), but want P (B |A).

Aria Nosratinia Probability and Statistics

Bayes theorem makes a seemingly hard problem very easy!

Variation on Bayes Rule

P (A|Bi )P (Bi ) n j =1 P (A|Bj )P (Bj )

Aria Nosratinia Probability and Statistics

Bayes Example (2)

Aria Nosratinia Probability and Statistics

Bayes Example (2)

Aria Nosratinia Probability and Statistics