L13: Revision of Probability & Random Processes

So far, we’ve modelled signals & channels as deterministic

phenomena. I.e. all signal values at any time instant can, in principle,
be determined in advance.
All channel non-idealities have also been deterministic, e.g. amplitude
& phase distortion, weak nonlinearity. However,

 Practical channels also suffer from unpredictable effects, e.g.

channel interference, multipath fading, thermal electronic noise in
amplifiers etc.
 Comm systems are designed without advance knowledge of the
precise message to be transported, but statistical properties of the
type of message (speech, music, video, data) may be available.

 Need to understand probability & random processes…

L13 1
 Revision of Probability
(P&S sec. 4.1)

• The fundamental concept in probability is a random experiment, 

the precise outcome  of which (an elementary event) cannot be 
predicted in advance.

• E.g.,  could be the thermal noise at the output of the receiver, or 

the bit error pattern in a received data frame.

• The sample space  is defined as the set of all possible 

outcomes,  : { possible}.

• Depending on the problem,  may be discrete, e.g. bit errors, or

• nondiscrete (continuous),e.g. thermal noise.

L13 2
 Probability Measures and Spaces
An event is defined as some subset S  . To every allowed
event S , we attach a probability P(S) [0,1] such that:

1. P()  1.

2. If S1 , S 2 , S3  are disjoint, i.e. P(Si  S j )  P(Si )  P(S j ),

 
i  j , then P i 1 Si  i 1 P ( Si )
 

Consequently, the probability measure P also satisfies:

 P(S )  1  P(S). This also implies that P()  0.
 P(S1  S2 )  P(S1 )  P(S2 )  P(S1  S2 )
 If S1  S 2 then P(S1 )  P(S 2 )

L13 3
 A Technicality

Let B denote the class of all valid events.

The triple (, B, P) is called a probability space.
We want B to have the following basic properties:
  is a valid event .
2. If S is a valid event then so is S . 
3. If S1 , S 2 , S3 , are valid events, then so is i 1 Si .

For discrete  , B is taken to be the class 2 of all subsets of  .

I.e., any subset of the sample space is a valid event.

However, for continuous  we usually have to restrict B so that it

is strictly smaller than 2 . I.e., not all subsets are valid events.

L13 4
Other Properties of Probability Measures

 Given that an event X has occurred, we can renormalise

the probability of an event Y via the conditional probability
P(Y  X )
P(Y | X ) : .
P(X )
Reflects the fact that knowledge of a related event should
change the probabilities we assign.

 X & Y called mutually independent if

P(Y | X )  P(Y ) , i.e. P( X  Y )  P( X )P(Y ) .

L13 5
  Bayes’ Rule:
P(X  Y ) P(Y | X )P(X )
P(X | Y )   .
P(Y ) P(Y )
Important in digital comms & signal processing, where X
corresponds to the input to a noisy channel or system
& Y corresponds to the output.)

L13 6
 Random Variables
A random variable (rv) X is a function on domain  .
Its range may be discrete (e.g., the number of bit errors in a data
frame) or continuous (e.g. the output of a noisy amplifier).
For simplicity, we’ll consider only rv’s X :   R.

By convention, the  argument is often dropped,

e.g.    : X ( )  x is just written {X  x}.
(some mild technical conditions on X (.) are usually assumed so that sets like
this constitute valid events)

Cumulative distribution function (cdf)

FX (x) : P   : X ()  x  P(X  x).

L13 7
E.g. of Sample Space & RV (Fig. 8.2-3, Carlson et.al)

L13 8
 Properties of CDF

1. 0  FX (x)  1, x .
2. 0  FX (x)  1 is nondecreasing with x .
3. lim FX (x)  1, lim FX (x)  0.
x x  
4. P(a  X  b)  FX (b)  FX (a)
5. P(X  b)  FX (b)  FX (b  )

A real-valued rv is called
 discrete, if its cdf has the shape of a staircase
 continuous, if its cdf is continuous everywhere
 mixed, if its cdf is continuous at some points but
discontinuous at others.

L13 9
E.g. of CDF (Fig. 8.2-3, Carlson et al)

L13 10
 Probability Density Function

For continuous rv’s with differentiable cdf, the probability density

d
function (pdf) is defined by f (x) : F (x)
X X
dx
1. f X (x)  0, x.
b
2.
 X
f (x)dx  FX (b)  P( X  b).
3. b
a

X f (x)dx  FX (b)  FX (a)  P(a  X  b).
4.

f X (x)dx  1.

For discrete rv’s, usually prefer probability mass function (pmf)

PX (x) : P(X  x) (1-4 still hold for pmf, with sums not integrals)

L13 11
 Averages & Moments


 Expected value or mean E[X ] :  xf X (x)dx : mX



 If Y  g( X ), then E[Y ]   g(x) f X (x)dx .

 E[aX  b]  aE[X ]  b. 
 Moment of (integer) order n, E[X ] :  x f X (x)dx
n n

 Variance  2 : E[(X  m ) 2 ]   (x  m ) 2 f (x)dx  E[ X 2 ]  m 2


X X  X X X

Standard deviation  X

L13 12
 Using PDF’s & PMF’s to Model Comms Systems

 Transmitting a bit through a noisy discrete channel which

changes it (01, 10) with probability p ….

 The number of bit errors in a data packet of size n

 The phase of a sinusoidal carrier wave…

 Thermal noise in an electronic circuit…

L13 13
 Multiple Random Variables

 Joint pmf FX ,Y (x, y) : P(X  x,Y  y)

 Joint pdf f X ,Y (x, y) :  2

FX ,Y (x, y)
xy
1. FX (x)  FX ,Y (x,).

2. f X (x)   f X ,Y (x, y)dy
FX ,Y (x, y)    f X ,Y (u,v)dvdu
x y
3.

f X ,Y (x, y)
 Conditional pdf fY | X ( y | x) 
f X (x)
 Mutual independence: fY | X ( y | x)  fY ( y) .

L13 14
 Joint Expectations & Moments

 

 E[g(X ,Y )]    g(x, y) f X ,Y (x, y)dxdy .

 Covariance cov[X ,Y ] : E[(X  mX )(Y  mY )]
 Correlation = E[ XY ] cov[X ,Y ]
: [1,1]
 Correlation coefficient  X ,Y
 XY
The correlation coefficient is a measure of the linear
dependency between two rvs. As it’s dimensionless, it isn’t
affected by a change of units, unlike cov.

X,Y are called uncorrelated if cov[X ,Y ]  0.

Independence  Uncorrelatedness, but not the reverse.

L13 15
 Transformation of RV’s

 Let Y  g( X ) ,
where g : R  R
is 1-1 & differentiable. Then

dy
fY ( y)  f X x 
dx

 Let Z  g( X ,Y ),W  h( X ,Y ) , where the mapping

(g, h) : R 2  R 2 is 1-1 & differentiable. Then
 xz z

f Z ,W ( z , w)  f X ,Y  x, y  | det J ( x, y ) | , where J ( x, y ) :  w y
w 
 x y 

L13 16
 Characteristic Functions


Characteristic function (cf) X ( ) : E[e
j X
]   e jX f X ( x) dx

Essentially a Fourier transform, so there’s a 1-1 relationship
between a pdf and its cf.

 If X ,Y independent, then X Y ( )  X ( )Y ( )

X ( )
n
1 d
 E[ X ]  n
n

d
n
j  0
These properties make it easier to obtain the moments of an rv or
the pdf of a sums of independent rv’s.

L13 17
 Gaussian PDF

1  ( x  m) 2 
f X ( x)  exp    0, x  R.
 2  2 2

Written X ~ N(m,  2 ).
N (0,1) is called the unit normal or Gaussian distribution.
Graph decays quickly to 0 for large | x  m |
& has upward convex bulge near m
 Characteristic bell - shape.
 ( x) : cdf of N(0,1).
Q( x) : 1   ( x), upper tail probability.

L13 18
 The Central Limit Theorem

Suppose X 1 , X 2 , are iid rv's with E[ X i ]   & var[X i ]   2

n

Let S n :  X i .
i 1
n n

 E[S n ]   E[X i ]  n , var[S n ]   var[X i ]  n 2 .

i 1 i 1

S n  n
Then define the mean zero, unit variance rv Z n : .
 n
z
CLT : lim P[Z n  z]  1  exp(x 2
/ 2)dx.
n 2


I.e. FZ n (z)  N(0,1) cdf , regardless of X distribution.

L13 19
 Jointly Gaussian RVs

Let X  [ X 1 ,, X n ]' be a random vector with

E[ X ]  m  R n , covariance matrix E[(X  m)( X  m)']  K  R nn .
X is called Gaussian, or equivalently X 1 ,, X m are jointly Gaussian, if

f X (x)  f X 1 ,, X n (x1 ,, xn ) 

exp  0.5(x  m)' K 1
(x  m) 
.
(2 ) | det K |
n/2 1/ 2

~ N(m, K ). Completely characterised by mean & covariance matrix.

Reduces to usual scalar case when m  1 with K   2 .

L13 20
 Properties of Joint Gaussians

 If X is a Gaussian random vector, then so is Y  AX  B

In particular, this means that if X 1 ,  X n are jointly Gaussian
rv’s, then any subset of them is also jointly Gaussian.

 If X ,Y are independent & Gaussian, then so is X  Y

 If X 1 , X n are jointly Gaussian and mutually uncorrelated rv’s,

then they must be mutually independent.

These properties make the Gaussian pdf extremely useful when

analysing the effects of noise in linear filters or channels.

L13 21