5.

Introduction to Probability Theory

The statistician is basically concerned with drawing conclusions (or inference)
from experiments involving uncertainties. For these conclusions and
inferences to be reasonably accurate, an understanding of probability theory
is essential.

In this section, we shall develop the concept of probability with equally likely
outcomes.

Experiment, Sample Space and Event

Experiment: This is any process of observation or procedure that:

(1) Can be repeated (theoretically) an infinite number of times; and

(2) Has a well-defined set of possible outcomes.

Sample space: This is the set of all possible outcomes of an experiment.

Event: This is a subset of the sample space of an experiment.

Consider the following illustrations:

Experiment 1: Tossing a coin.

Sample space: S = {Head or Tail} or we could write:

S = {0, 1} where 0 represents a tail and  1 represents a head.

Experiment 2: Tossing a coin twice.

Sample Space:

S = {HH, TT, HT, TH} where

H represents head and

T represents tail.
Some possible events:
E1 = {Head},

E2 = {Tail},

E3 = {All heads}

Experiment 3: Throwing a die.

Sample space:

S = {1, 2, 3, 4, 5, 6} or S = {Even, odd}

Some events:

Even numbers, E1 = {2, 4, 6}

Odd numbers, E2 = {1, 3, 5}

The number 1, E3 = {1}

At least 3, E4 = {3, 4, 5, 6}

Experiment 4: Defective items

Two items are picked, one at a time, at random from a manufacturing process, and each item is
inspected and classified as defective or non-defective.

Sample space:

S = {NN, ND, DN, DD} where

N = Non-defective

D = Defective

Some events:

E1 = {only one item is defective} = {ND, DN}

E2 = {Both are non-defective} = {NN}

RANDOM EXPERIMENT:
 An experiment which produces different results even though it is repeated a large number of times under
essentially similar conditions is called a Random Experiment.

The tossing of a fair coin, the throwing of a balanced die, drawing of a card from a well-shuffled deck of 52 playing cards,
selecting a sample, etc. are examples of random experiments.

A random experiment has three properties:

i) The experiment can be repeated, practically or theoretically, any number of times.

ii )The experiment always has two or more possible outcomes. An experiment that has only one possible outcome, is not a
random experiment.

iii )The outcome of each repetition is unpredictable, i.e. it has some degree of uncertainty.

Considering a more realistic example, interviewing a person to find out whether or not he or she is a smoker is an example of
a random experiment. This is so because this example fulfils all the three properties that have just been discussed:

1.This process of interviewing can be repeated a large number of times.

2.To each interview, there are at least two possible replies: ‘I am a smoker’ and ‘I am not a smoker’.

3.For any interview, the answer is not known in advance i.e. there is an element of uncertainty regarding the person’s reply.

A concept that is closely related with the concept of a random experiment is the concept of the Sample Space.

SAMPLE SPACE:

A set consisting of all possible outcomes that can result from a random experiment (real or conceptual), can be
defined as the sample space for the experiment and is denoted by the letter S.

Each possible outcome is a member of the sample space, and is called a sample point in that space.

Let us consider a few examples:

EXAMPLE-1

The experiment of tossing a coin results in either of the two possible outcomes: a head (H) or a tail (T).

(We assume that it is not possible for the coin to land on its edge or to roll away.)

The sample space for this experiment may be expressed in set notation as S = {H, T}.

‘H’ and ‘T’ are the two sample points.

EXAMPLE-2

The sample space for tossing two coins once (or tossing a coin twice) will contain four possible outcomes denoted
by

S = {HH, HT, TH, TT}.

In this example, clearly, S is the Cartesian product A ´ A, where A = {H, T}.

EXAMPLE-3

The sample space S for the random experiment of throwing two six-sided dice can be described by the Cartesian
product A ´ A, where
A = {1, 2, 3, 4, 5,6}.

In other words,
S=A´A

= {(x, y) | x Î A and y Î A}

where x denotes the number of dots on the upper face of the first die, and y denotes the number of dots on the upper face of
the second die.

Hence, S contains 36 outcomes or sample points, as shown below:

S = { (1, 1), (1, 2), (1, 3), (1, 5), (1, 6),

(2, 1), (2, 2), (2, 3), (2, 5), (2, 6),

(3, 1), (3, 2), (3, 3), (3, 5), (3, 6),

(4, 1), (4, 2), (4, 3), (4, 5), (4, 6),

(5, 1), (5, 2), (5, 3), (5, 5), (5, 6),

(6, 1), (6, 2), (6, 3), (6, 5), (6, 6) }

The next concept is that of events:

EVENTS:

Any subset of a sample space S of a random experiment, is called an event.

In other words, an event is an individual outcome or any number of outcomes (sample points) of a random experiment.

SIMPLE & COMPOUND EVENTS:

An event that contains exactly one sample point, is defined as a simple event. A compound event contains more
than one sample point, and is produced by the union of simple events.

EXAMPLE

The occurrence of a 6 when a die is thrown, is a simple event, while the occurrence of a sum of 10 with a pair of
dice, is a compound event, as it can be decomposed into three simple events (4, 6), (5, 5) and (6, 4).

OCCURRENCE OF AN EVENT:

An event A is said to occur if and only if the outcome of the experiment corresponds to some element of A.

EXAMPLE:

Suppose we toss a die, and we are interested in the occurrence of an even number.

If ANY of the three numbers ‘2’, ‘4’ or ‘6’ occurs, we say that the event of our interest has occurred.

In this example, the event A is represented by the set

{2, 4, 6}, and if the outcome ‘2’ occurs, then, since this outcome is corresponding to the first element of the set A, therefore,
we say that A has occurred.

COMPLEMENTARY EVENT:

The event “not-A” is denoted by `A or Ac and called the negation (or complementary event) of A.

EXAMPLE:

If we toss a coin once, then the complement of “heads” is “tails”.

 If we toss a coin four times, then the complement of “at least one head” is “no heads”.

A sample space consisting of n sample points can produce 2n different subsets (or simple and compound events).

EXAMPLE:

Consider a sample space S containing 3 sample points, i.e. S = {a, b, c}.

Then the 23 = 8 possible subsets are

f, {a}, {b}, {c}, {a, b},

{a, c}, {b, c}, {a, b, c}

Each of these subsets is an event.

The subset {a, b, c} is the sample space itself and is also an event. It always occurs and is known as the certain or sure event.

The empty set f is also an event, sometimes known as impossible event, because it can never occur.

MUTUALLY EXCLUSIVE EVENTS:

Two events A and B of a single experiment are said to be mutually exclusive or disjoint if and only if they cannot both occur at
the same time i.e. they have no points in common.

EXAMPLE-1:

When we toss a coin, we get either a head or a tail, but not both at the same time.

The two events head and tail are therefore mutually exclusive.

EXAMPLE-2:

Consider two coins that we toss at the same time. The occurrence of either a Head or a Tail on one of these doesn’t affect the
probability of the occurrence of H or T of the other coin.

EXAMPLE-3:

When a die is rolled, the events ‘even number’ and ‘odd number’ are mutually exclusive as we can get either an even number
or an odd number in one throw, not both at the same time. Similarly, a student either qualifies or fails, a single birth must be
either a boy or a girl, it cannot be both, etc., etc. Three or more events originating from the same experiment are mutually
exclusive if pair wise they are mutually exclusive. If the two events can occur at the same time, they are not mutually
exclusive, e.g., if we draw a card from an ordinary deck of 52 playing cars, it can be both a king and a diamond.

Therefore, kings and diamonds are not mutually exclusive.

Similarly, inflation and recession are not mutually exclusive events.

Speaking of playing cards, it is to be remembered that an ordinary deck of playing cards contains 52 cards arranged in 4 suits
of 13 each. The four suits are called diamonds, hearts, clubs and spades; the first two are red and the last two are black.

The face values called denominations, of the 13 cards in each suit are ace, 2, 3, …, 10, jack, queen and king.

The face cards are king, queen and jack.

These cards are used for various games such as whist, bridge, poker, etc.

We have discussed the concepts of mutually exclusive events.

Another important concept is that of exhaustive events.

EXHAUSTIVE EVENTS:
Events are said to be collectively exhaustive, when the union of mutually exclusive events is equal to the entire sample space
S.

EXAMPLES:

1. In the coin-tossing experiment, ‘head’ and ‘tail’ are collectively exhaustive events.

2. In the die-tossing experiment, ‘even number’ and ‘odd number’ are collectively exhaustive events.

In conformity with what was discussed in the last lecture:

PARTITION OF THE SAMPLE SPACE:

A group of mutually exclusive and exhaustive events belonging to a sample space is called a partition of the sample
space. With reference to any sample space S, events A and `A form a partition as they are mutually exclusive and their union
is the entire sample space. The Venn Diagram below clearly indicates this point.

S A

A is shaded

Next, we consider the concept of equally likely events:

EQUALLY LIKELY EVENTS:

Two events A and B are said to be equally likely, when one event is as likely to occur as the other.

In other words, each event should occur in equal number in repeated trials.

EXAMPLE:

When a fair coin is tossed, the head is as likely to appear as the tail, and the proportion of times each side is
expected to appear is 1/2.

EXAMPLE:

If a card is drawn out of a deck of well-shuffled cards, each card is equally likely to be drawn, and the probability
that any card will be drawn is 1/52

THE AXIOMATIC DEFINITION OF PROBABILITY:

 This definition, introduced in 1933 by the Russian mathematician Andrei N. Kolmogrov, is
based on a set of AXIOMS.

Let S be a sample space with the sample points E1, E2, … Ei, …En. To each sample point, we assign a real number, denoted
by the symbol P(Ei), and called the probability of Ei, that must satisfy the following basic axioms:

Axiom 1:

For any event Ei,

0 < P(Ei) < 1.

Axiom 2:

P(S) =1

for the sure event S.

Axiom 3:

If A and B are mutually exclusive events (subsets of S), then

P (A È B) = P(A) + P(B).

EXAMPLE :

Table-1

below shows the numbers of births in England and Wales in 1956 classified by (a) gender and (b) whether liveborn or
stillborn.

Table-1

Number of births in England and Wales in 1956 by sex and whether live- or still born.

(Source Annual Statistical Review)

There are four possible events in this double classification:

• Male livebirth (denoted by A),

• Male stillbirth (denoted by B),
• Female livebirth (denoted by C)
and

• Female stillbirth (denoted by D),

•
The relative frequencies corresponding to the figures of Table-1 are given in Table-2:

Table-2

Proportion of births in England and Wales in 1956 by gender and whether live- or stillborn.

(Source Annual Statistical Review)

The total number of births is large enough for these relative frequencies to be treated for all practical purposes as
PROBABILITIES.

Let us denote the compound events ‘Male birth’ and ‘Stillbirth’ by the letters M and S.Now a male birth
occurs whenever either a male livebirth or a male stillbirth occurs, and so the proportion of male birth, regardless of whether
they are live-or stillborn, is equal to the sum of the proportions of these two types of birth; that is to say,

p(M) = p(A or B) = p(A) + p(B)

= .5021 + .0120 = .5141

Similarly, a stillbirth occurs whenever either a male stillbirth or a female stillbirth occurs and so the proportion of stillbirths,
regardless of sex, is equal to the sum of the proportions of these two events:

p(S) = p(B or D) = p(B) + p(D)

= .0120 + .0109 = .0229.

Let us now consider some basic LAWS of probability.

These laws have important applications in solving probability problems.

LAW OF COMPLEMENTATION:

If `A is the complement of an event A relative to the sample space S, then

P ( Ā )=1−P ( A ) .
Hence the probability of the complement of an event is equal to one minus the probability of the event.

Complementary probabilities are very useful when we are wanting to solve questions of the type ‘What is the
probability that, in tossing two fair dice, at least one even number will appear?’

EXAMPLE:

A coin is tossed 4 times in succession. What is the probability that at least one head occurs?

(1) The sample space S for this experiment consists of 24 = 16 sample points (as each toss can result in 2 outcomes),and
(2) we assume that each outcome is equally likely.

If we let A represent the event that at least one head occurs, then A will consist of MANY sample points, and
the process of computing the probability of this event will become somewhat cumbersome! So, instead of denoting this
particular event by A, let us denote its complement i.e. “No head” by A.

Thus the event A consists of the SINGLE sample point {TTTT}.

Therefore P(A ) = 1/16.

Hence by the law of complementation, we have

1 15
P ( Ā )=1−P ( A ) =1− = .
16 16
The next law that we will consider is the Addition Law or the General Addition Theorem of Probability:

ADDITION LAW:

If A and B are any two events defined in a sample space S, then

P(AÈB) = P(A) + P(B) – P(AÇB)

In words, this law may be stated as follows:

“If two events A and B are not mutually exclusive, then the probability that at least one of them occurs, is given by the sum
of the separate probabilities of events A and B minus the probability of the joint event A Ç B.”