Random Processes

Gaurav S. Kasbekar
Dept. of Electrical Engineering
IIT Bombay
 Motivation
• For a study of the noise performance of communication
systems, we model noise as a random process:
❑noise often modeled as Additive White Gaussian Noise
• Also, concept of power spectral density (PSD) useful for
frequency domain analysis of random processes
❑e.g., message signals and noise
• Later, we will see that PSD of a wide-sense stationary random
process is Fourier transform of its autocorrelation function
• So now we study:
❑random processes,
❑stationarity, strict-sense stationary and wide-sense stationary
processes
❑mean function and autocorrelation function
❑power spectral density
❑Gaussian random processes
❑white noise processes
 Review of Random Vectors
• Consider a random vector 𝑿 = (𝑋1 , 𝑋2 , … , 𝑋𝑛 )
• Joint CDF:
❑ 𝐹𝑋1,…,𝑋𝑛 𝑥1 , … , 𝑥𝑛 = 𝑃 𝑋1 ≤ 𝑥1 , … , 𝑋𝑛 ≤ 𝑥𝑛
• Joint PDF:
𝜕𝑛 𝐹𝑋1 ,…,𝑋𝑛 𝑥1 ,…,𝑥𝑛
❑ 𝑓𝑋1,…,𝑋𝑛 𝑥1 , … , 𝑥𝑛 = 𝜕𝑥1 …𝜕𝑥𝑛
❑ 𝑃 (𝑋1 , 𝑋2 , … , 𝑋𝑛 ) ∈ 𝐷 :
❑ ‫𝑋𝑓 𝐷׬‬1 ,…,𝑋𝑛 𝑥1 , … , 𝑥𝑛 𝑑𝑥1 … 𝑑𝑥𝑛
• 𝑋1 , … , 𝑋𝑛 independent if:
❑ the events {𝑋1 ∈ 𝐴1 }, … , {𝑋𝑛 ∈ 𝐴𝑛 } are independent for all 𝐴1 ⊆ ℛ, …, 𝐴𝑛 ⊆ ℛ
• 𝑋1 , … , 𝑋𝑛 independent iff:
1) 𝐹𝑋1,…,𝑋𝑛 𝑥1 , … , 𝑥𝑛 = 𝐹𝑋1 𝑥1 … 𝐹𝑋𝑛 𝑥𝑛
2) 𝑓𝑋1,…,𝑋𝑛 𝑥1 , … , 𝑥𝑛 = 𝑓𝑋1 𝑥1 … 𝑓𝑋𝑛 𝑥𝑛 (continuous case)
• Mean vector of the random vector 𝑿 = (𝑋1 , 𝑋2 , … , 𝑋𝑛 ) :
❑ 𝝁𝑿 = (𝜇1 , 𝜇2 , … , 𝜇𝑛 ), where 𝜇𝑗 = 𝐸(𝑋𝑗 )
• Covariance matrix:
❑ K 𝑿 : 𝑛 × 𝑛 matrix with (𝑖, 𝑗)’th element 𝐶𝑖,𝑗 = Cov(𝑋𝑖 , 𝑋𝑗 )= 𝐸 𝑋𝑖 − 𝜇𝑖 𝑋𝑗 − 𝜇𝑗
• Properties of covariance matrix K 𝑿 of a random vector 𝑿 = (𝑋1 , 𝑋2 , … , 𝑋𝑛 ):
❑ K 𝑿 is a symmetric matrix
❑ K 𝑿 = 𝐸 (𝑿 − 𝝁𝑿 )(𝑿 − 𝝁𝑿 )𝑇
❑ K 𝑿 is positive-semidefinite
 Recall
• Consider a probability space (Ω, ℱ, 𝑃)
• A random variable 𝑋:
❑ for each ω ∈ Ω, 𝑋(ω) a real number
• A random vector 𝑋1 , … , 𝑋𝑛 :
❑for each ω ∈ Ω, 𝑋1 (ω), … , 𝑋𝑛 (ω) a real vector
• Next: a random process 𝑋(𝑡)
❑for each ω ∈ Ω, 𝑋(𝑡, ω) is a real function
 Random Process
• Random process is a function from Ω to set of real
functions
• For fixed 𝑡, 𝑋(𝑡) is:
❑ a random variable
• For fixed 𝑡 and ω, 𝑋(𝑡, ω) is:
❑ a real number

Ref: Haykin, Chapter 1

 Examples
• AC voltage from wall socket measured starting
from a random instant
❑ 𝑋 𝑡 = 𝑅 cos 𝜔𝑡 + Θ
❑ Phase Θ often modeled as uniform in [−𝜋, 𝜋]
❑ Amplitude 𝑅 and frequency 𝜔 also random variables
(they fluctuate in a small range)
• Thermal noise in a resistor
❑random motion of electrons produces small current
❑𝑋 𝑡 : noise current in given direction at time 𝑡
• Signal at receiver in communication system
❑transmitted data symbols unknown
❑modeled as random process (e.g., sequence of
symbols with unknown phases and amplitudes)
 Examples (contd.)
• E.g. of signal at receiver in communication system
❑Each pulse of width 𝑇, amplitude 𝐴 or −𝐴 with probability
½ each, independently of other pulses
❑𝑡𝑑 : uniform in [0, 𝑇]

Ref: Haykin, Chapter 1

 Distribution
• Recall: for fixed 𝑡, 𝑋(𝑡) is a random variable
❑distribution specified by 𝐹𝑋 𝑡 𝑥 = 𝑃(𝑋(𝑡) ≤ 𝑥)
❑ or by PDF of 𝑋(𝑡)
• More generally, for times 𝑡1 , … , 𝑡𝑛 , joint distribution
of 𝑋(𝑡1 ), … , 𝑋(𝑡𝑛 ) specified by:
❑𝐹𝑋 𝑡1 ,…,𝑋 𝑡𝑛 𝑥1 , … , 𝑥𝑛 = 𝑃(𝑋(𝑡1 ) ≤ 𝑥1 , … , 𝑋(𝑡𝑛 ) ≤
𝑥𝑛 )
❑ or by joint PDF of 𝑋(𝑡1 ), … , 𝑋(𝑡𝑛 )
• Distribution of a random process 𝑋(𝑡) specified by
𝐹𝑋 𝑡1 ,…,𝑋 𝑡𝑛 𝑥1 , … , 𝑥𝑛 for all 𝑛 ≥ 1, 𝑡1 , … , 𝑡𝑛 and
all 𝑥1 , … , 𝑥𝑛
 Distribution (contd.)
• Distribution of a random process 𝑋(𝑡) specified by
𝐹𝑋 𝑡1 ,…,𝑋 𝑡𝑛 𝑥1 , … , 𝑥𝑛 for all 𝑛 ≥ 1, 𝑡1 , … , 𝑡𝑛 and
all 𝑥1 , … , 𝑥𝑛
• E.g. for random process on previous slide:
❑if 𝑡𝑘 − 𝑡𝑘−1 > 𝑇 for 𝑘 = 2, … , 𝑛, then
𝐹𝑋 𝑡1 ,…,𝑋 𝑡𝑛 𝑥1 , … , 𝑥𝑛 :
o 𝐹𝑋 𝑡1 𝑥1 … 𝐹𝑋 𝑡𝑛 𝑥𝑛
o 𝐺 𝑥1 … 𝐺 𝑥𝑛 , where 𝐺(. ) is the CDF of a random variable
that takes values 𝐴 and −𝐴 w.p. ½ each

Ref: Haykin, Chapter

 Averages
• 𝐹𝑋 𝑡1 ,…,𝑋 𝑡𝑛 𝑥1 , … , 𝑥𝑛 for all 𝑛, 𝑡1 , … , 𝑡𝑛 and
𝑥1 , … , 𝑥𝑛 contain detailed information
• But in several applications, some averages are
enough
• Recall
❑mean and variance of a random variable
❑mean vector and covariance matrix of random
vector
• We now define similar quantities for a random
process
 Mean, Autocovariance and
• Mean function: Autocorrelation Functions
❑ 𝜂𝑋 𝑡 = 𝐸(𝑋 𝑡 )
• Autocovariance function:
❑ 𝐾𝑋 𝑡1 , 𝑡2 = 𝐸(𝑋(𝑡1 )𝑋(𝑡2 )) − 𝜂𝑋 𝑡1 𝜂𝑋 𝑡2
❑ measure of correlation between 𝑋(𝑡1 ) and 𝑋(𝑡2 ) when
var(𝑋 𝑡 ) is constant
o as is the case for a stationary process
cov 𝑋,𝑌
o Note: Pearson’s correlation coefficient: 𝜌𝑋,𝑌 =
𝜎𝑋 𝜎𝑌
• Autocorrelation function:
❑ 𝑅𝑋 𝑡1 , 𝑡2 = 𝐸(𝑋(𝑡1 )𝑋(𝑡2 ))
❑ 𝐾𝑋 𝑡1 , 𝑡2 = 𝑅𝑋 𝑡1 , 𝑡2 when 𝐸(𝑋 𝑡 ) = 0 for all 𝑡
o often the case for signals in communication systems since dc component
can cause problems in circuitry (e.g., saturation of amplifiers)
• 𝑅𝑋 (𝑡, 𝑡):
❑ 𝐸(𝑋 𝑡 2 )
❑ average instantaneous power at time 𝑡
 Positive Semi-definiteness
• Recall
❑var(𝑋) ≥ 0 for every random variable 𝑋
❑𝐾𝑿 is positive semi-definite for every random vector 𝑿
• Definition: A function 𝑔(𝑡, 𝑠) is positive semi-
definite if for all 𝑁 > 0, all 𝑡1 < 𝑡2 < ⋯ < 𝑡𝑁 and
all constants 𝑎1 , 𝑎2 , … 𝑎𝑁 , we have:
❑σ𝑁 σ 𝑁
𝑖=1 𝑗=1 𝑎𝑖 𝑎𝑗 𝑔( 𝑡𝑖 , 𝑡𝑗 ) ≥ 0
• Claim: The autocovariance function of every
random process 𝑋 𝑡 is positive semi-definite
❑ follows from fact that covariance matrix of random
vector with components 𝑋 𝑡1 , … , 𝑋(𝑡𝑛 ) is positive
semi-definite
• Claim: The autocorrelation function of every
random process 𝑋 𝑡 is positive semi-definite
 Example
• 𝑋 𝑡 = 𝑅 cos 𝜔𝑡 + Θ
• 𝜔 constant, Θ and 𝑅 independent r.v.s
• Θ uniform in [−𝜋, 𝜋]
• 𝜂𝑋 𝑡 :
❑𝐸 𝑅 𝐸 cos 𝜔𝑡 + Θ
❑0
• 𝑅𝑋 𝑡1 , 𝑡2 :
❑ 𝐸 𝑅 cos 𝜔𝑡1 + Θ 𝑅 cos 𝜔𝑡2 + Θ
1
❑ 𝐸 𝑅2 𝐸 cos 𝜔(𝑡1 − 𝑡2 )+ cos 𝜔𝑡1 + 𝜔𝑡2 + 2Θ
2
1
❑ 𝐸(𝑅2 ) cos 𝜔(𝑡1 − 𝑡2 )
2
 Example
• Each pulse of width 𝑇, amplitude 𝐴 or −𝐴 with
probability ½ each, independently of other pulses
• 𝑡𝑑 : uniform in [0, 𝑇]
• Exercise: Show that 𝑅𝑋 𝑡𝑖 , 𝑡𝑗 equals:
|𝑡𝑗 −𝑡𝑖 |
𝐴2 1− , 𝑡𝑗 − 𝑡𝑖 < 𝑇,
❑ቐ 𝑇
0, else.

Ref: Haykin, Chapter 1

 Stationary Processes
• For several random processes that arise in practice,
statistics of 𝑋(𝑡) don’t change with time
• Actual values change, but probabilities of taking
different values don’t change
• E.g.:
1) Thermal noise in a resistor
❑ 𝐼(𝑡): current at time 𝑡
2) AC voltage from wall socket measured starting from a
random instant
❑assume amplitude and frequency constant for simplicity
❑ 𝑋 𝑡 = cos 𝜔𝑡 + Θ , where phase Θ uniform in [−𝜋, 𝜋]
3) Light particle suspended in a liquid
❑collides with liquid molecules and moves randomly
❑observed starting from a random instant
❑𝑋(𝑡): 𝑥 coordinate of particle at time 𝑡
 Strict Sense Stationarity
• Recall: distribution of 𝑋(𝑡) described by
𝐹𝑋 𝑡1 ,…,𝑋 𝑡𝑛 𝑥1 , … , 𝑥𝑛
• 𝑋𝑐 𝑡 = 𝑋(𝑡 + 𝑐): shifted process
• 𝑋(𝑡) is SSS if all joint distributions of 𝑋(𝑡) and
𝑋𝑐 𝑡 same for every 𝑐
• Definition: 𝑋(𝑡) is SSS if
𝐹𝑋 𝑡1 ,…,𝑋 𝑡𝑛 𝑥1 , … , 𝑥𝑛 =
𝐹𝑋 𝑡1+𝑐 ,…,𝑋 𝑡𝑛+𝑐 𝑥1 , … , 𝑥𝑛 for all 𝑐, 𝑛,
𝑡1 , … , 𝑡𝑛 , and 𝑥1 , … , 𝑥𝑛
 𝜂𝑋 𝑡 and 𝑅𝑋 𝑡1 , 𝑡2 for a SSS Process
• 𝑋(𝑡): a SSS process
• 𝜂𝑋 𝑡 :
❑𝐸 𝑋(𝑡)
❑𝐸 𝑋(𝑡 − 𝑡) = 𝐸 𝑋(0)
❑some constant independent of 𝑡, say 𝜂
• 𝑅𝑋 𝑡1 , 𝑡2 :
❑𝐸(𝑋(𝑡1 )𝑋(𝑡2 ))
❑𝐸 𝑋(𝑡1 − 𝑡1 𝑋(𝑡2 − 𝑡1 )) = 𝐸 𝑋(0 𝑋(𝑡2 − 𝑡1 ))
❑ depends only on difference 𝑡2 − 𝑡1
❑ say, 𝑅𝑋 𝜏 , where 𝜏 = 𝑡2 − 𝑡1
 Wide Sense Stationarity
• Definition: 𝑋(𝑡) is WSS if 𝜂𝑋 𝑡 is
independent of 𝑡 and 𝑅𝑋 𝑡1 , 𝑡2 depends only
on 𝑡2 − 𝑡1
• If 𝑋(𝑡) is SSS, then it is WSS
• Converse not true in general
• 𝑋 𝑡 = 𝐴 cos 𝜔0 𝑡 + 𝐵 sin 𝜔0 𝑡 Example
• 𝐴, 𝐵: i.i.d. random variables with mean 0, variance 𝜎 2 , and third
moment 𝐸 𝐴3 = 𝐸 𝐵 3 = 𝜆 ≠ 0
• 𝜂𝑋 𝑡 :
❑ 𝐸 𝑋(𝑡)
❑ 𝐸 𝐴 cos 𝜔0 𝑡 + 𝐵 sin 𝜔0 𝑡 = 𝐸(𝐴) cos 𝜔0 𝑡 + 𝐸(𝐵) sin 𝜔0 𝑡
❑0
• 𝑅𝑋 𝑡1 , 𝑡2 :
❑ 𝐸(𝑋(𝑡1 )𝑋(𝑡2 ))
❑ 𝐸( 𝐴 cos 𝜔0 𝑡1 + 𝐵 sin 𝜔0 𝑡1 𝐴 cos 𝜔0 𝑡2 + 𝐵 sin 𝜔0 𝑡2 )
❑ 𝐸൫𝐴2 cos 𝜔0 𝑡1 cos 𝜔0 𝑡2 + 𝐴𝐵 cos 𝜔0 𝑡1 sin 𝜔0 𝑡2
+𝐴𝐵 sin 𝜔0 𝑡1 cos 𝜔0 𝑡2 + 𝐵2 sin 𝜔0 𝑡1 sin 𝜔0 𝑡2 )
❑ 𝐸 𝐴2 cos 𝜔0 𝑡1 cos 𝜔0 𝑡2 + 𝐸 𝐴𝐵 cos 𝜔0 𝑡1 sin 𝜔0 𝑡2 +
𝐸 𝐴𝐵 sin 𝜔0 𝑡1 cos 𝜔0 𝑡2 + 𝐸(𝐵2 ) sin 𝜔0 𝑡1 sin 𝜔0 𝑡2
❑ 𝜎 2 cos 𝜔0 𝑡1 cos 𝜔0 𝑡2 + 0 + 0 + 𝜎 2 sin 𝜔0 𝑡1 sin 𝜔0 𝑡2
❑ 𝜎 2 cos 𝜔0 (𝑡2 −𝑡1 )
• 𝐸 𝑋(𝑡)3 :
❑ 𝜆 [cos 3 𝜔0 𝑡 + sin3 𝜔0 𝑡 ] (exercise)
• So 𝑋(𝑡) is WSS, but not SSS
 Example
• 𝑋 𝑡 = cos 𝜔𝑡 + Θ
• 𝜔 constant, Θ uniform in [−𝜋, 𝜋]
• Recall: this is WSS since mean is 0 and 𝑅𝑋 𝑡1 , 𝑡2 =
1
cos 𝜔(𝑡1 − 𝑡2 )
2
• Is 𝑋 𝑡 SSS?
❑Yes
• 𝑋 𝑡 + 𝑐 = cos 𝜔𝑡 + (𝜔𝑐 + Θ)
• So 𝑋 𝑡 + 𝑐 = cos 𝜔𝑡 + Θ𝑐 , where Θ𝑐 = 𝜔𝑐 + Θ
❑Θ𝑐 : uniform in [−𝜋 + 𝜔𝑐, 𝜋 + 𝜔𝑐]
• 𝐹𝑋 𝑡1 +𝑐 ,…,𝑋 𝑡𝑛 +𝑐 𝑥1 , … , 𝑥𝑛 :
❑ 𝑃(cos 𝜔𝑡1 + Θ𝑐 ≤ 𝑥1 , … , cos 𝜔𝑡𝑛 + Θ𝑐 ≤ 𝑥𝑛 )
❑ independent of 𝑐 (exercise)
 Example
• Each pulse of width 𝑇, amplitude 𝐴 or −𝐴 with
probability ½ each, independently of other pulses
• 𝑡𝑑 : uniform in [0, 𝑇]
• Recall: this is a WSS process with:
|𝜏|
𝐴2 1− , 𝜏 < 𝑇,
❑𝑅𝑋 𝜏 = ቐ 𝑇
0, else.
• Exercise: It is also SSS

Ref: Haykin, Chapter 1

 Properties of 𝑅𝑋 𝜏
• Recall: 𝑅𝑋 𝜏 = 𝐸 𝑋 𝑡 𝑋(𝑡 + 𝜏)
1) 𝑅𝑋 0 :
❑ 𝐸(𝑋 𝑡 2 )
❑ average power
2) 𝑅𝑋 𝜏 = 𝑅𝑋 −𝜏 ∀𝜏
❑ Proof: 𝑅𝑋 −𝜏 = 𝐸 𝑋(𝑡 − 𝜏)𝑋 𝑡 = 𝐸 [𝑋(𝑡 ′ )𝑋(𝑡 ′ +
𝜏)] = 𝑅𝑋 𝜏 , where 𝑡 ′ = 𝑡 − 𝜏
3) |𝑅𝑋 𝜏 | ≤ 𝑅𝑋 0 ∀𝜏
❑ Proof: follows from Cauchy-Schwartz inequality
(exercise)
 Speed of Fluctuation of Process
• 𝑅𝑋 𝜏 = 𝐸(𝑋 𝑡 𝑋 𝑡 + 𝜏 )
• Correlation between 𝑋(𝑡) and 𝑋(𝑡 + 𝜏) when
𝐸(𝑋 𝑡 ) = 0 for all 𝑡 and var(𝑋 𝑡 ) constant
• Roughly indicates how rapidly 𝑋(𝑡) fluctuates
❑e.g., in fig. on left, process in upper (resp., lower) fig. is
slowly (resp., rapidly) fluctuating

Ref: Haykin, Chapter 1

 Power Spectral Density
• Note: Our current focus is on WSS random
processes; we are not presently concerned with
non-stationary processes
• Informally, density of average power, 𝐸 𝑋 𝑡 2 , at
frequency 𝑓
• Similar to energy spectral density for deterministic
signals
 Energy Spectral Density
• Let 𝑥(𝑡) be a deterministic signal
• Total energy of 𝑥 𝑡 :
∞
❑‫׬‬−∞ 𝑥(𝑡)2 𝑑𝑡
❑assume that this is finite
• By Parseval’s theorem:
∞ ∞
❑‫׬‬−∞ 𝑥(𝑡)2 𝑑𝑡 = ‫׬‬−∞ |𝑋 𝑓 |2 𝑑𝑓,
❑where, 𝑋 𝑓 is the Fourier transform of 𝑥(𝑡)
• |𝑋 𝑓 |2 is the density of energy at frequency 𝑓
❑justification: pass 𝑥(𝑡) through a narrow band-pass
filter centered at 𝑓0 and of bandwidth ∆𝑓
• |𝑋 𝑓 |2 called ESD of signal 𝑥(𝑡)
∞
❑specifies how the energy ‫׬‬−∞ 𝑥(𝑡)2 𝑑𝑡 is distributed at
various frequencies
 Power Spectral Density: Motivation
• Can we define ESD for a WSS process 𝑋 𝑡 ?
• Expected total energy of 𝑋 𝑡 :
∞
❑𝐸 ‫׬‬−∞ 𝑋 𝑡 2 𝑑𝑡
∞ 2 ∞
❑‫׬‬−∞ 𝐸 𝑋 𝑡 𝑑𝑡 (assuming 𝐸(. ) and ‫׬‬−∞.
can be
swapped)
❑= ∞ (assuming average power is not zero)
• WSS process is a power signal, not an energy signal
• PSD used instead of ESD
❑density of average power, 𝐸(𝑋 𝑡 2 ), at frequency 𝑓
 Power Spectral Density
• Definition: 𝑆𝑋 (𝑓) is Fourier transform of
𝑅𝑋 𝜏
∞
• 𝑆𝑋 𝑓 = ‫׬‬−∞ 𝑅𝑋 𝜏 𝑒 −𝑗2𝜋𝑓𝜏 𝑑𝜏
• 𝑅𝑋 𝜏 can be obtained from 𝑆𝑋 𝑓 by inverse
Fourier transform or tables
• Interpretation: 𝑆𝑋 𝑓 is density of average
power, 𝐸(𝑋 𝑡 2 ), at frequency 𝑓
 Example
• Recall: autocorrelation function of the random
process shown in fig.:
|𝜏|
𝐴2 1− , 𝜏 < 𝑇,
𝑅𝑋 𝜏 = ൞ 𝑇
0, else.
• 𝑆𝑋 𝑓 :
❑ 𝐴2 𝑇sinc 2 𝑓𝑇

Ref: Haykin, Chapter 1

 Example
• 𝑋 𝑡 = 𝐴 cos 2𝜋𝑓𝑐 𝑡 + Θ
• Θ uniform in [−𝜋, 𝜋]
𝐴2
• Recall: 𝑅𝑋 𝜏 = cos 2𝜋𝑓𝑐 𝜏
2
• 𝑆𝑋 𝑓 :
𝐴2
❑ 𝛿 𝑓 − 𝑓𝑐 + 𝛿 𝑓 + 𝑓𝑐
4
 Properties of 𝑆𝑋 𝑓
1) 𝑆𝑋 𝑓 is real and even
❑ Proof: Fourier transform of real and even function
is real and even
2 ∞
2) 𝐸 𝑋 𝑡 = ‫׬‬−∞ 𝑆𝑋 𝑓 𝑑𝑓
❑Proof: put 𝜏 = 0 in the equation 𝑅𝑋 𝜏 =
∞
‫׬‬−∞ 𝑆𝑋 𝑓 𝑒 𝑗2𝜋𝑓𝜏 𝑑𝑓
❑consistent with interpretation that 𝑆𝑋 𝑓 is density
of average power, 𝐸(𝑋 𝑡 2 ), at frequency 𝑓
 WSS Process Through LTI Filter
• 𝑋 𝑡 : WSS process with mean 𝜂𝑋 and
autocorrelation function 𝑅𝑋 𝜏
• Input to LTI system with impulse response ℎ(𝑡)
• Is output 𝑌 𝑡 WSS?
❑ we will show: yes

Ref: Haykin, Chapter 1

 𝜂𝑌 (𝑡) and 𝑅𝑌 𝑡, 𝑡 + 𝜏
∞
• 𝑌 𝑡 = ‫׬‬−∞ ℎ 𝜏 𝑋 𝑡 − 𝜏 𝑑𝜏
• 𝜂𝑌 (𝑡):
∞
❑ 𝜂𝑋 ‫׬‬−∞ ℎ 𝜏 𝑑𝜏
❑ independent of 𝑡
❑ 𝑅𝑌 𝑡, 𝑡 + 𝜏 :
❑ 𝐸(𝑌 𝑡 𝑌 𝑡 + 𝜏 )
∞ ∞
❑ 𝐸 ‫׬‬−∞ ℎ 𝜏1 𝑋 𝑡 − 𝜏1 𝑑𝜏1 ‫׬‬−∞ ℎ 𝜏2 𝑋 𝑡 + 𝜏 − 𝜏2 𝑑𝜏2
∞ ∞
❑ ‫׬‬−∞ ‫׬‬−∞ ℎ 𝜏1 ℎ 𝜏2 𝐸 𝑋 𝑡 − 𝜏1 𝑋 𝑡 + 𝜏 − 𝜏2 𝑑𝜏1 𝑑𝜏2
∞ ∞
❑ ‫׬‬−∞ ‫׬‬−∞ ℎ 𝜏1 ℎ 𝜏2 𝑅𝑋 𝜏 + 𝜏1 − 𝜏2 𝑑𝜏1 𝑑𝜏2
∞ ∞
❑ ‫׬‬−∞ ℎ 𝜏1 ‫׬‬−∞ ℎ 𝜏2 𝑅𝑋 𝜏 + 𝜏1 − 𝜏2 𝑑𝜏2 𝑑𝜏1
∞
❑ ‫׬‬−∞ ℎ 𝜏1 𝑔 𝜏 + 𝜏1 𝑑𝜏1 , where 𝑔 𝜏 = 𝑅𝑋 𝜏 ∗ ℎ(𝜏)
❑ 𝑔(𝜏) ∗ ℎ(−𝜏)
❑ 𝑅𝑋 𝜏 ∗ ℎ(𝜏) ∗ ℎ(−𝜏)
❑ independent of 𝑡
• So 𝑌 𝑡 is WSS
 PSD of 𝑌 𝑡
• 𝑅𝑌 𝜏 = 𝑅𝑋 𝜏 ∗ ℎ(𝜏) ∗ ℎ(−𝜏)
• Let 𝐻 𝑓 be Fourier transform of ℎ(𝑡)
• 𝑆𝑌 𝑓 :
❑𝑆𝑋 𝑓 𝐻 𝑓 𝐻 ∗ (𝑓)
❑𝑆𝑋 𝑓 |𝐻 𝑓 |2
 Interpretation of PSD
• Want to show that 𝑆𝑋 𝑓0 is density of average power, 𝐸(𝑋 𝑡 2 ), at
frequency 𝑓0
• Pass 𝑋 𝑡 through appropriate filter:
❑ ideal band-pass filter of small width ∆𝑓 around 𝑓0
∆𝑓 ∆𝑓
1, 𝑓0 − ≤ |𝑓| ≤ 𝑓0 + ,
❑ 𝐻 𝑓 =ቐ 2 2
0, else.
• 𝑆𝑌 𝑓 :
∆𝑓 ∆𝑓
𝑆 𝑓 , 𝑓0 − ≤ |𝑓| ≤ 𝑓0 + ,
❑ ቐ 𝑋 2 2
0, else.
• Output 𝑌 𝑡 consists of components from spectrum of 𝑋 𝑡 only in
frequency band of width ∆𝑓 around 𝑓0 and −𝑓0
• 𝐸 𝑌 𝑡 2 :
∞
❑ ‫׬‬−∞ 𝑆𝑌 𝑓 𝑑𝑓
∆𝑓
𝑓0 + 2
❑ 2‫׬‬ ∆𝑓 𝑆𝑋 𝑓 𝑑𝑓
𝑓0 − 2
❑ ≈ 2𝑆𝑋 𝑓0 ∆𝑓 (assuming that 𝑆𝑋 𝑓 is continuous)
• Thus, 𝑆𝑋 𝑓0 is density of avg. power, 𝐸(𝑋 𝑡 2 ), at frequency 𝑓0
 Non-Negativity of PSD
• 𝑆𝑋 𝑓0 ≥ 0 for all 𝑓0
• Proof:
❑ 𝑌 𝑡 as in previous slide
❑ 𝐸 𝑌 𝑡 2 ≈ 2𝑆𝑋 𝑓0 ∆𝑓 ≥ 0
 Gaussian Processes: Motivation
• Thermal noise (Johnson–Nyquist noise) in
resistor often modeled as Gaussian process
❑noise that arises due to random motion of
electrons in resistor, regardless of any applied
voltage

Ref: Haykin, Chapter 1

 Gaussian Process: Definition
• Recall: 𝑋1 , 𝑋2 , … , 𝑋𝑛 are jointly Gaussian (or
equivalently, 𝑿 = (𝑋1 , … , 𝑋𝑛 ) is a Gaussian
random vector) if:
1 −1
𝑓𝑿 𝒙 = exp (𝑿 − 𝝁𝑿 )𝑇 K 𝑿 −1 (𝑿 − 𝝁𝑿 ) ,
(2𝜋)𝑛/2 det(K𝑿 ) 2

❑ where 𝑿 = (𝑋1 , 𝑋2 , … , 𝑋𝑛 )
• Recall: if 𝑿 = (𝑋1 , … , 𝑋𝑛 ) is a Gaussian random
vector and 𝒂 = (𝑎1 , … , 𝑎𝑛 ) ≠ 𝟎, then 𝑌 = 𝒂𝑇 𝑿 is
a Gaussian random variable
• Definition: 𝑋(𝑡) is a Gaussian random process if
for all 𝑛 ≥ 1 and 𝑡1 , … , 𝑡𝑛 , 𝑋(𝑡1 ), … , 𝑋(𝑡𝑛 ) are
jointly Gaussian
 Properties
• Recall: distribution of 𝑋(𝑡) specified by
𝐹𝑋 𝑡1 ,…,𝑋 𝑡𝑛 𝑥1 , … , 𝑥𝑛 = 𝑃(𝑋(𝑡1 ) ≤ 𝑥1 , … , 𝑋(𝑡𝑛 ) ≤
𝑥𝑛 )
1) Distribution of a Gaussian process 𝑋(𝑡) completely
determined by 𝜂𝑋 𝑡 and 𝑅𝑋 𝑡1 , 𝑡2
❑ Proof: follows from fact that distribution of Gaussian random vector
completely determined by mean vector and autocovariance matrix
❑ 𝐹𝑋 𝑡1 ,…,𝑋 𝑡𝑛 𝑥1 , … , 𝑥𝑛 completely specified by 𝝁𝑿 and K 𝑿 , where
𝑿 = (𝑋(𝑡1 ), 𝑋(𝑡2 ), … , 𝑋(𝑡𝑛 )); also, 𝑅𝑋 𝑡𝑖 , 𝑡𝑗 = 𝐾𝑋 𝑡𝑖 , 𝑡𝑗 +
𝝁𝑿 𝑡𝑖 𝝁𝑿 𝑡𝑗
2) If a Gaussian process 𝑋(𝑡) is WSS, then it is also SSS
❑ Proof: since 𝑋(𝑡) is WSS, 𝜂𝑿 𝑡 = 𝜂𝑿 (say) and 𝑅𝑋 𝑡1 , 𝑡2 = 𝑅𝑋 𝜏
(say), where 𝜏 = 𝑡2 − 𝑡1
❑ Let 𝑌 𝑡 = 𝑋(𝑡 + 𝑐); then 𝜂𝒀 𝑡 = 𝜂𝑿 and 𝑅𝑌 𝑡1 , 𝑡1 + 𝜏 =
𝐸 𝑌 𝑡1 𝑌 𝑡1 + 𝜏 = 𝐸 𝑋 𝑡1 + 𝑐 𝑋 𝑡1 + 𝑐 + 𝜏 = 𝑅𝑋 𝜏
 Properties (contd.)
• Recall: 𝑋1 , … , 𝑋𝑛 are jointly Gaussian iff 𝑎1 𝑋1 + ⋯ +
𝑎𝑛 𝑋𝑛 is a Gaussian random variable for all 𝑎1 , … , 𝑎𝑛
• Generalization to Gaussian process:
3) 𝑋(𝑡) is a Gaussian process iff for all functions 𝑔(𝑡)
and 𝑇1 < 𝑇2 :
𝑇2
𝑌𝑔 = ‫𝑔 𝑇׬‬𝑡 𝑋 𝑡 𝑑𝑡 is a Gaussian random variable
1
whenever 𝐸 𝑌𝑔 2 < ∞
❑ Proof of “if” part: Suffices to show: 𝑋(𝑡1 ), … , 𝑋(𝑡𝑛 ) are jointly
Gaussian for arbitrary 𝑡1 , … , 𝑡𝑛
❑ In turn, suffices to show: for given constants 𝑡1 , … , 𝑡𝑛 and 𝑎1 , … , 𝑎𝑛 ,
σ𝑛𝑖=1 𝑎𝑖 𝑋 𝑡𝑖 is a Gaussian random variable
❑ Select some 𝑇1 < 𝑡1 , 𝑇2 > 𝑡𝑛 , and let 𝑔 𝑡 = σ𝑛𝑖=1 𝑎𝑖 𝛿 𝑡 − 𝑡𝑖
𝑇2
❑ Then 𝑌𝑔 = ‫𝑔 𝑇׬‬ 𝑡 𝑋 𝑡 𝑑𝑡 = σ𝑛𝑖=1 𝑎𝑖 𝑋 𝑡𝑖 is a Gaussian random
1
variable
❑ So 𝑋(𝑡1 ), … , 𝑋(𝑡𝑛 ) are jointly Gaussian
Applications of Gaussian Processes
• Many physical processes are approximately
Gaussian
• Reasons:
1) A large-scale phenomenon often arises from the
combination of a large number of small-scale
i.i.d. phenomena
2) Central Limit Theorem (CLT)
• 𝑋(𝑡): noise current Thermal Noise: Intuition
• 𝑋 𝑡 = σ𝑁 𝑖=1 𝑋𝑖 (𝑡),
❑ where 𝑋𝑖 (𝑡): noise current due to random motion of
electron 𝑖
• 𝑋1 𝑡 , … , 𝑋𝑁 (𝑡) approximately independent and
identically distributed
• Want to show: 𝑋 𝑡 approximately a Gaussian process
• For fixed 𝑡, 𝑋 𝑡 approximately a Gaussian random
variable by CLT
• Next, fix 𝑔(𝑡) and 𝑇1 < 𝑇2
𝑇2
• 𝑌𝑔 = ‫𝑔 𝑇׬‬ 𝑡 𝑋 𝑡 𝑑𝑡 :
1
𝑇2
❑ σ𝑁
𝑖=1 ‫𝑔 𝑇׬‬ 𝑡 𝑋𝑖 𝑡 𝑑𝑡 (substituting for 𝑋 𝑡 and assuming
1
integral and summation can be interchanged)
❑ approximately Gaussian by CLT
 Gaussian Process Through LTI System
• Claim: If a Gaussian process, 𝑋(𝑡), is input to LTI
system, then output 𝑌(𝑡) is a Gaussian process
• Proof:
∞
❑𝑌 𝑡 = ‫׬‬−∞ 𝑋 𝜏 ℎ 𝑡 − 𝜏 𝑑𝜏
❑Fix a function 𝑔(𝑡) and 𝑇1 < 𝑇2
𝑇2
❑Let 𝑌𝑔 = ‫𝑔 𝑇׬‬ 𝑡 𝑌 𝑡 𝑑𝑡
1
𝑇 ∞
❑Then 𝑌𝑔 = ‫ 𝑇׬‬2 𝑔 𝑡 ‫׬‬−∞ 𝑋 𝜏 ℎ 𝑡 − 𝜏 𝑑𝜏 𝑑𝑡
1
∞ 𝑇2
❑𝑌𝑔 = ‫׬‬−∞ 𝑋 𝜏 ‫ 𝑇׬‬ℎ 𝑡 − 𝜏 𝑔 𝑡 𝑑𝑡 𝑑𝜏
1
∞ 𝑇
❑𝑌𝑔 = ‫׬‬−∞ 𝑋 𝜏 𝑔(𝜏)𝑑𝜏,
෤ where 𝑔෤ 𝜏 = ‫ 𝑇׬‬2 ℎ 𝑡 − 𝜏 𝑔 𝑡 𝑑𝑡
1

❑So 𝑌𝑔 is a Gaussian random variable

 White Noise Process: Motivation
• Recall: thermal noise in resistor
❑noise that arises due to random motion of
electrons
• Modeled as a white noise process
• Let 𝑋(𝑡): thermal noise current White Noise Process
• Since motion of electrons highly random, intuitively we expect 𝑋(𝑡𝑖 ) and 𝑋(𝑡𝑗 ) to
be roughly uncorrelated even for small 𝑡𝑖 − 𝑡𝑗
• So 𝑅𝑋 (𝜏) ≈ 0 for all 𝜏, except values close to 0
• Consider an idealized model of above:
𝑁
❑ 𝑅𝑋 𝜏 = 20 𝛿(𝜏),
❑ where 𝑁0 > 0 is a constant
• 𝑆𝑋 𝑓 :
𝑁0
❑ for all 𝑓
2
𝑁0
• Definition: A WSS process 𝑋(𝑡) is white noise if 𝜂𝑋 = 0 and 𝑅𝑋 𝜏 = 𝛿(𝜏)
2

Ref: Haykin, Chapter 1

 White Noise Process (contd.)
• Note: “White” is by analogy to white light, which
has equal amounts of all visible band frequencies
• Technical Note: Consider an alternative idealized
model:
𝑁0
, 𝜏 = 0,
❑𝑅𝑋 𝜏 = ൝2
0, 𝜏 ≠ 0.
❑Reason for not using above alternative model:
oaverage instantaneous power is 0 since 𝑆𝑋 𝑓 = 0 ∀𝑓
 Average Instantaneous Power
• Consider a white noise process 𝑋(𝑡) with 𝑅𝑋 𝜏 =
𝑁0
𝛿(𝜏), where 𝑁0 > 0
2
• Average instantaneous power of 𝑋 𝑡 :
❑∞
• Thus, white noise process does not exist in reality
• Still, thermal noise modeled as white noise process
 Reason for Modeling Thermal Noise as
White Noise Process
• In practice, can’t observe thermal noise 𝑋(𝑡) directly
• Can observe output of band-limited filter whose input
is thermal noise
❑even if observed directly on spectrum analyzer, filtering
occurs due to finite bandwidth
• Experimentally, it is found that if 𝑋 𝑡 is input to filter
with freq. response 𝐻(𝑓), then output 𝑌 𝑡 has PSD:
𝑁
❑𝑆𝑌 𝑓 ≈ 0 |𝐻 𝑓 |2 , where 𝑁0 is a constant
2
❑above found to hold for all frequencies that arise in
practice
𝑁0
❑consistent with the model 𝑆𝑋 𝑓 = for all 𝑓
2
• Thus, white noise is a simple model that agrees with
experiments
 Additive White Gaussian Noise
• Recall: thermal noise modeled as a Gaussian process
as well as white noise
• The above motivates the concept of “white
Gaussian noise” (formal definition on next slide)
• More generally, noise in communication systems
widely modeled as AWGN
❑“additive” since assumed to add to other effects such as
fading, interference, non-linearity, etc.
• Generated by a number of sources:
❑e.g., thermal noise, shot noise, black body radiation
 Technical Note
• Let 𝑋(𝑡): a white Gaussian noise process
• 𝐸(𝑋 𝑡 2 ):
∞ ∞ 𝑁0
❑𝑅𝑋 0 = ‫׬‬−∞ 𝑆𝑋 𝑓 𝑑𝑓 = ‫׬‬−∞ 𝑑𝑓 = ∞
2
• Since variance of sample is ∞, technically not a
Gaussian process
• Definition: A process 𝑋(𝑡) is WGN if
❑ it is a white noise process and
𝑇2
❑ for all functions 𝑔(𝑡) and 𝑇1 < 𝑇2 , 𝑌𝑔 = ‫𝑔 𝑇׬‬ 𝑡 𝑋 𝑡 𝑑𝑡 is
1
a Gaussian random variable whenever 𝐸 𝑌𝑔 2 <∞
• If a WGN process is input to an LTI system, then
under some technical conditions, output is a
Gaussian process (exercise)
 Example: Low-Pass Filtered WGN
𝑁0
• 𝑋(𝑡): a white Gaussian noise process with PSD
2
• Input to ideal LPF with frequency response 𝐻(𝑓), where:
1, 𝑓 < 𝐵,
❑|𝐻 𝑓 | = ቊ
0, else.
• Let 𝑌(𝑡) be output process
• 𝑆𝑌 𝑓 :
❑𝑆𝑋 𝑓 |𝐻 𝑓 |2
𝑁0
, 𝑓 < 𝐵,
❑ቐ 2
0, else.
• 𝑅𝑌 𝜏 :
❑𝑁0 𝐵sinc(2𝐵𝜏)
• Also, 𝑌(𝑡) is a Gaussian process (see exercise on previous
slide)
 Example: 𝑅𝐶 Low-Pass Filtered White Noise
𝑁0
• 𝑤(𝑡): White Gaussian noise of zero mean and PSD
2
• Applied to a low-pass 𝑅𝐶 filter as shown in fig
• Want PSD and autocorrelation function of noise
process, say 𝑛(𝑡), at output of filter
• 𝑆𝑛 (𝑓):
𝑁0 1
❑𝑆𝑤 (𝑓)|𝐻 𝑓 |2 , where 𝑆𝑤 𝑓 = and 𝐻 𝑓 =
2 1+𝑗2𝜋𝑓𝑅𝐶
𝑁0 /2
❑
1+(2𝜋𝑓𝑅𝐶)2
• 𝑅𝑛 (𝜏):
𝑁0 |𝜏|
❑ exp −
4𝑅𝐶 𝑅𝐶

Ref: “Communication Systems” by Haykin and Moher

 Cross-Correlation Function
• Consider two random processes 𝑋(𝑡) and 𝑌(𝑡) with
autocorrelation functions 𝑅𝑋 (𝑡, 𝑢) and 𝑅𝑌 𝑡, 𝑢 ,
respectively
• Cross-correlation function of 𝑋(𝑡) and 𝑌(𝑡) defined
to be:
❑ 𝑅𝑋𝑌 𝑡, 𝑢 = 𝐸 𝑋 𝑡 𝑌(𝑢)
• 𝑋(𝑡) and 𝑌(𝑡) said to be jointly wide-sense
stationary if:
❑they are both WSS and
❑𝑅𝑋𝑌 𝑡, 𝑢 depends only on the difference 𝜏 = 𝑡 − 𝑢
Example: Quadrature Modulated Processes
• Suppose 𝑋(𝑡) is a WSS process and let:
❑𝑋1 𝑡 = 𝑋 𝑡 cos(2𝜋𝑓𝑐 𝑡 + Θ),
❑𝑋2 𝑡 = 𝑋 𝑡 sin(2𝜋𝑓𝑐 𝑡 + Θ),
❑where Θ is uniformly distributed over (0,2𝜋) and is
independent of 𝑋(𝑡)
• Are 𝑋1 𝑡 and 𝑋2 𝑡 jointly WSS?
• Are 𝑋1 𝑡 and 𝑋2 𝑡 WSS?
❑Yes (exercise)
• 𝑅𝑋1 𝑋2 (𝑡, 𝑢):
❑𝐸 𝑋1 𝑡 𝑋2 𝑢
1
❑− 𝑅𝑋 𝜏 sin(2𝜋𝑓𝑐 𝜏), where 𝜏 = 𝑡 − 𝑢 (exercise)
2
• So 𝑋1 𝑡 and 𝑋2 𝑡 are jointly WSS