Chapter 5

Limits Theorem

The most important theoretical results in probability theory are limit theorems. Of these, the
most important are Laws of large numbers and Central limit theorems.

Definition 5.0.119 (Markov’s inequality). If X is a random variable that takes only non-
negative values, then, for any value a > 0,

E[X]
P(X ≥ a) ≤ .
a

Definition 5.0.120 (Chebyshev’s inequality). If X is a random variable with finite mean µ

and variance σ 2 , then, for any value k > 0,

σ2
P(|X − µ| ≥ k) ≤ .
k2

Remark 5.0.121. The importance of Markov’s and Chebyshev’s inequalities is that they enable
us to derive bounds on probabilities when only the mean, or both the mean and the variance
are known.

Example 5.0.122. If V ar(X) = 0, then P (X = E[X]) = 1. In other words, the only random
variables having variances equal to 0 are those which are constant with probability 1.

Definition 5.0.123 (Sample mean). Let X be a random variable for which the mean, E[X] =
µ, is unknown. Let X1 , X2 , .... denotes n independent repeated measurements of X; that is, the
Xj0 s are independent, identically distributed (iid) random variables with the same pdf as X. The
sample mean of the sequence is
n
1X
Mn = Xj
n i=1

σ2
Remark 5.0.124. The expected value and variance of the sample mean are µ and n
respectively.

18
CHAPTER 5. LIMITS THEOREM

Example 5.0.125. Let X1 , X2 , .... denotes n independent, identically distributed (iid) random
variables, each with mean µ and variance σ 2 . If Mn denotes the sample mean then prove that

σ2
P(|Mn − µ| < ) ≥ 1 − .
n2

Example 5.0.126. A voltage of constant, but unknown, value is to be measured. Each mea-
surement Xj is actually the sum of the desired voltage v and a noise voltage Nj of zero mean
and standard deviation of 1 microvolt. Assume that the noise voltages are independent ran-
dom variables. How many measurements are required so that the probability that Mn is within
 = 1microvolt of the true mean is at least 0.99?

Definition 5.0.127 (The weak law of large numbers). Let X1 , X2 , .... be a sequence of
independent and identically distributed random variables, each having finite mean E[Xi ] = µ.
Then, for any  > 0,   
 X1 + X2 .... + Xn 
P  − µ ≥  → 0.
n

Definition 5.0.128 (The central limit theorem). Let X1 , X2 , .... be a sequence of indepen-
dent and identically distributed random variables, each having mean µ and variance σ 2 . Then
the distribution of
X1 + X2 .... + Xn − nµ
√
σ n
tends to normal as n → ∞. That is, for −∞ < a < ∞,
  a
X1 + X2 .... + Xn − nµ
Z
1 2 /2
P √ ≤a →√ e−x dx as n → ∞.
σ n 2π −∞

Remark 5.0.129. The amazing part about the central limit theorem is that the summands Xj
can have any distribution as long as they have a finite mean and finite variance.This gives the
result its wide applicability.

Definition 5.0.130 (The strong law of large numbers). Let X1 , X2 , .... be a sequence of
independent and identically distributed random variables, each having finite mean E[Xi ] = µ.
Then, with probability 1,

X1 + X2 .... + Xn
→ µ as n → ∞.
n

i.e.  
X1 + X2 .... + Xn
P lim =µ = 1.
n→∞ n

Definition 5.0.131 (Vector space represntation of Random Variables). The set of all

19
 CHAPTER 5. LIMITS THEOREM

random variables defined on a sample space form a vector space with respect to addition and
scalar multiplication.

Definition 5.0.132 (Linear Independence). Let X1 , X2 , .....XN be the sequence of random

variables. If
c1 X1 + c2 X2 + ..... + cN XN = 0

implies that c1 = c2 = .... = cN = 0 then X1 , X2 , .....XN are linearly independent.

Definition 5.0.133 (Inner Product). The inner product between random variable X and Y
is defined as Z ∞ Z ∞
hX, Y i = E[XY ] = xyfX,Y (x, y)dxdy
−∞ −∞

Definition 5.0.134 (Orthogonal Random Variable). Two random variable X and Y are
said to be orthogonal if E[XY ] = 0.

Remark 5.0.135. For zero-mean random variables,

Orthogonality ⇔ uncorrelatedness

Definition 5.0.136 (Sure convergence). The sequence of random variables{Xn (ζ)} converges
surely to the random variable X(ζ) if the sequence of functions Xn (ζ)converges to the function
Xn (ζ) as n → ∞ for all ζ in S:

lim Xn (ζ) = X( ζ) for all ζ ∈ S

n→∞

Definition 5.0.137 (Almost sure (a.s.) convergence). The sequence of random variables{Xn (ζ)}
converges almost surely to the random variable X(ζ) if the sequence of functions Xn (ζ)converges
to the function Xn (ζ) as n → ∞ for all ζ in S,except possibly on a set of probability zero; that
is, h i
P ζ : lim Xn (ζ) = X( ζ) = 1.
n→∞

In almost-sure convergence some of the sample sequences may not converge, but these must
all belong to ζ‘s that are in a set that has probability zero. The strong law of large numbers
is an example of almost-sure convergence. Note that sure convergence implies almost-sure
convergence.

Example 5.0.138. Let ζ be selected at random from the interval S = [0, 1], where we assume
that the probability that is ζ in a subinterval of S is equal to the length of the subinterval. For
ζ
n = 1, 2, .... we define the following sequences of random variables:(a) Un (ζ) = n
(b) Vn (ζ) =
1 n −n(nζ−1)
ζ(1 − n
) (c) Wn (ζ) = ζe (d) Yn (ζ) = cos(2πnζ) (e) Zn (ζ) = e . Which of these sequences
converge surely? almost surely? Identify the limiting random variable.

20
CHAPTER 5. LIMITS THEOREM

Definition 5.0.139 (Mean square convergence). The sequence of random variables{Xn (ζ)}
converges in the mean square sense to the random variable X(ζ) if

E[(Xn (ζ) − X(ζ))2 ] → 0 as n → ∞.

We denote mean square convergence by (limit in the mean)

l.i.m.Xn (ζ) = X(ζ) as n → ∞.

Mean square convergence is of great practical interest in electrical engineering applications

because of its analytical simplicity and because of the interpretation of E[(Xn − X)2 ] as the
“power” in an error signal.

Example 5.0.140. Let Xn be a sequence of iid random variables with mean µ and variance
σ 2 < ∞. Let Mn be the associated sequence of arithmetic averages,
n
1X
Mn = Xj
n i=1

Show that Mn converges to µ in the mean square sense.

21