Fundamentals of

Information Theory
Lecture 1. Introduction
Prof. CHEN Jie
Lab. 201, School of EIE
BeiHang University
 Teaching Staff

Dr. YU Ze

Office: F617, NMB

Email: yz613@buaa.edu.cn

Dr. SUN Bing

Office: F617, NMB

Email: bingsun@buaa.edu.cn

Prof. CHEN Jie

Dean of Depart. of Info.& Com. Eng., SEIE Dr. HUANG Qin
Office: F615, New Main Building (NMB) Office: F509, NMB
Email: chenjie@buaa.edu.cn Email: qinhuang@buaa.edu.cn
Copyright©BUAA201
Copyright©BUAA201
Text Book

Thomas M. Cover (1938–2012)

Fellow Members of the IEEE
IEEE R. W. Hamming Medal Recipients

Copyright©BUAA201
Copyright©BUAA201
Lecture Notes

Copyright©BUAA201
Copyright©BUAA201
Laboratory Manual

Copyright©BUAA201
Copyright©BUAA201
 Course Website
http://infortheory.buaa.edu.cn/

Copyright©BUAA201
Copyright©BUAA201
 Outline
1．Introduction and Preview
2．Entropy and Mutual Information
3．Asymptotic Equipartition Property
4．Markov chains
5．Data Compression
6．Channel Capacity
7．Differential Entropy
8．Gaussian Channel
9．Maximum Entropy and Spectral Estimation
10．Rate Distortion Theory
11．Network Information Theory

Copyright©BUAA201
Copyright©BUAA201
1. Introduction to Information
How to distinguish
information, signal
And message
What is
Information？

Theoretical model of
a typical communication
system ？

Copyright©BUAA201
Copyright©BUAA201
 1. Introduction to Information
Information, in its general sense, is
“Knowledge communicated or received concerning a parti
-cular fact or circumstance. "
 Information can’t be predicted and resolves uncertainty.
 The uncertainty of an event is measured by its probability
of occurrence and is inversely proportional to that.
 The more uncertain an event is, the more information is
required to resolve uncertainty of that event.
 The amount of information is measured in bits.
Example:
 information in fair one coin flip: log2(2/1) = 1 bit
 whereas in fair two coin flip is log2(4/1) = 2 bits..
http://en.wikipedia.org/
Copyright©BUAA201
Copyright©BUAA201
 1. Introduction to Information
Information, in its most restricted technical sense, is
a sequence of symbols that can be interpreted as a
message.
 Information can be recorded as signs, or transmitted
as signals.
 Information is any kind of event that affects the stat
e of a dynamic system.
 Information is the message being conveyed.
 Information is closely related to notions of constraint
, communication, control, data, instruction, knowled
ge, meaning, understanding, mental stimuli, pattern,
perception, representation, and entropy.

http://en.wikipedia.org/
Copyright©BUAA201
Copyright©BUAA201
 1.1 Concept of information
What is information ?

fair one coin flip

S={T,F}

operator will receive a call in next oneCopyright©BUAA201

hour
Copyright©BUAA201
1.1 Concept of information

Variations in value of 10 resistor

Electromagnetic Interference

Noisy Channel
transmission

Copyright©BUAA201
Copyright©BUAA201
1.1 Concept of information

To be or not to be, that is the question.

William Shakespeare's play Hamlet

http://en.wikipedia.org/
Copyright©BUAA201
Copyright©BUAA201
1.1 Concept of information

Can you give some

more examples?
Copyright©BUAA201
Copyright©BUAA201
1.1 Concept of information

Can we measure
information?
Copyright©BUAA201
Copyright©BUAA201
 1.1 Concept of information

The logarithmic connection between

entropy and probability was first stated by
L. Boltzmann in his kinetic theory of gases
The famous formula for entropy S

S  k loge W

k = 1.3806505(24) × 10−23 J/K,

Boltzmann's constant
W is the Wahrscheinlichkeit, the frequency of
occurrence of a macrostate, more precisely, the
number of possible microstates corresponding
to the macroscopic state of a system L. Boltzmann(1844-1906)

http://en.wikipedia.org/
Copyright©BUAA201
Copyright©BUAA201
 1.1 Concept of information

Nyquist’s logarithm law (1924)

Harry Nyquist (1889–1976)

http://en.wikipedia.org/
Copyright©BUAA201
Copyright©BUAA201
 1.1 Concept of information

Hartley’s law (1928)

Ralph Hartley (1888–1970)

http://en.wikipedia.org/
Copyright©BUAA201
Copyright©BUAA201
 1.1 Concept of information
The uncertainty measure

Uncertainty   log p ( x)
The average uncertainty, Entropy

H ( X )   p ( x) log p ( x)
x

Claude Shannon (1916-2001)

Atheists /Electrical engineers
Mathematicians & Statisticians
Computer pioneers…………
IEEE Medal of Honor recipients

http://en.wikipedia.org/
Copyright©BUAA201
Copyright©BUAA201
1.1 Concept of information

Claude Shannon (1916-2001)

Atheists /Electrical engineers
Mathematicians & Statisticians
Computer pioneers…………
IEEE Medal of Honor recipients

http://en.wikipedia.org/
Copyright©BUAA201
Copyright©BUAA201
 1.1 Concept of information

What is Information:
Information causes
change;
If it doesn’t, it isn’t
information”

Claude Shannon (1916-2001)

http://en.wikipedia.org/
Copyright©BUAA201
Copyright©BUAA201
1.2 Timeline of information theory

Copyright©BUAA201
Copyright©BUAA201
1.2 Timeline of information theory

Copyright©BUAA201
Copyright©BUAA201
1.2 Timeline of information theory

Copyright©BUAA201
Copyright©BUAA201
1.2 Timeline of information theory

Copyright©BUAA201
Copyright©BUAA201
1.2 Timeline of information theory

Copyright©BUAA201
Copyright©BUAA201
 1.3 Information, Message and Signals
 Information: The uncertainty of source transmitted
by communication system, which is contained by
message and is still an abstract conception
 Message: More specific concept with all kinds of
forms such as language, symbol, image…… which
can be understood by both sides of communication
system, or can be acquired/processed/stored by an
information systems, e.g. remote sensing, GNSS….
 Signal: The most physical concept, which is carrier
of message, being measurable, visible and physical
Copyright©BUAA201
Copyright©BUAA201
1.3 Information, Message and Signals
Earth Observation
System Configuration

http://s4.sinaimg.cn/mw690/
Copyright©BUAA201
Copyright©BUAA201
 Example1.3.1 VHF Band-Apollo-17/ALSE

The Apollo 17 moon craft launched VHF radar antenna

by U.S.(Dec.1972) made the SAR
firstly perform in the space

 This SAR was named as Apollo

Lunar Sounder Experiment (ALSE)

 ALSE was the first application in

the human history to study the
Moon's surface and interior using
SAR based on the space probe

http://www.jpl.nasa.gov/
Copyright©BUAA201
Copyright©BUAA201
Example1.3.2 VHF Band- MARS Express

http://www.jpl.nasa.gov/
Copyright©BUAA201
Copyright©BUAA201
 Example1.3.3 S Band Cassini–Huygens

Saturn
Radar image: Titan North Pole Lakes http://en.wikipedia.org/
Copyright©BUAA201
Copyright©BUAA201
 Example1.3.5 Shuttle Radar Topography Mission

Shuttle Radar Topography

Mission (STRM,U.S.) use two
radar antenna on board the
space shuttle to implement
the single-pass SAR
interferometry Demonstration of STRM

Demonstration of interferogram acquired by

http://www.jpl.nasa.gov/ INSAR processing
Copyright©BUAA201
Copyright©BUAA201
Example1.3.5 Shuttle Radar Topography Mission

http://www.jpl.nasa.gov/
Copyright©BUAA201
Copyright©BUAA201
Example1.3.6 SAR image: DEM of volcano Etna

http://www.jpl.nasa.gov/
Copyright©BUAA201
Copyright©BUAA201
Example1.3.9 TerraSAR-X

http://www.dlr.de/
Copyright©BUAA201
Copyright©BUAA201
Example1.3.9 TerraSAR-X

http://www.dlr.de/
Copyright©BUAA201
Copyright©BUAA201
Example1.3.9 TerraSAR-X

http://www.dlr.de/
Copyright©BUAA201
Copyright©BUAA201
Example1.3.9 TerraSAR-X

http://www.dlr.de/
Copyright©BUAA201
Copyright©BUAA201
Example1.3.9 TerraSAR-X

http://www.dlr.de/
Copyright©BUAA201
Copyright©BUAA201
Example1.3.9 TerraSAR-X

http://www.dlr.de/
Copyright©BUAA201
Copyright©BUAA201
Example1.3.9 TerraSAR-X

http://www.dlr.de/
Copyright©BUAA201
Copyright©BUAA201
Microwave EO Satellites

TerraSAR-X
• Interferometric tandem
• X band
• Resolution 1m~18m

RADARSAT-2
• Polarimetric radar
• C band
• Resolution 1m-100m

Copyright©BUAA201
Copyright©BUAA201
 Example1.3.10 IKONOS optical satellite

 IKONOS is one of the most

advanced commercial optical
satellites.
 IKONOS played an
important role in the modern
warfare and military
application

Copyright©BUAA201
Copyright©BUAA201
Example1.3.10 IKONOS image of Beijing

Copyright©BUAA201
Copyright©BUAA201
Optical EO Satellites

IKONOS
• Resolution: Pan=1 m M
S(B,G,R,NIR)=4 m
• Scale: 1: 5,000
• Mono and stereo

GeoEye-1
• Resolution:
Pan=0.41/0.5m
MS(B,G,R,NIR)=1.6/2 m
• Scale: 1: 2,000
• Mono and stereo

Copyright©BUAA201
Copyright©BUAA201
 Optical EO Satellites

Space Imaging's IKONOS satellite captured these one-meter resolution colour

images of the World Trade Center before and after the terrorist attack
Copyright©BUAA201
Copyright©BUAA201
Optical EO Satellites
QuickBird-2
• Resolution: Pan=0.65 m
MS(B,G,R,NIR)= 2.62 m
• Scale: 1: 5,000
• Mono only

WorldView-1
• Resolution: Pan=0.5 m
• Scale: 1: 2,000
• Mono and stereo

WorldView-2
 Resolution: Pan=0.5 m
MS1(B,G,R,NIR) &
MS2(CB,Y,RE,NIR2)=2 m
 Scale: 1: 2,000
 Mono and stereo
Copyright©BUAA201
Copyright©BUAA201
Optical EO Satellites

Pléiades-1/2
Commercial June, 2012
• Resolution:
• Pan=0.7/0.5m
• MS(B,G,R,NIR)=2.8/2m
• Scale: 1: 2 000
• Mono and stéréo

WorldView-3 (2014)
Resolution:
• Pan= 0.30/0.5m

• Scale: 1: 2,000
• Mono and stereo
• 16bands
• 4 additional SWIR
bands
Copyright©BUAA201
Copyright©BUAA201
Example1.3.11 Terahertz image

Copyright©BUAA201
Copyright©BUAA201
Example1.3.11 Terahertz image

Copyright©BUAA201
Copyright©BUAA201
1.3 Communication system model

Channel

Copyright©BUAA201
Copyright©BUAA201
1.3 Communication system model

Copyright©BUAA201
Copyright©BUAA201
 1.3 Communication system model
Channel
p=1/3
Sound of ring bell Received sound (simulated)

Channel
p=1/3

Satellite Remote sensing image of Received image (simulated)

NMB, Beihang University
http://map.google.com Copyright©BUAA201
Copyright©BUAA201
1.3 Communication system model

Binary Symmetric
Channel
source p=0.01

Binary Symmetric
Channel
source p=0.1

Binary Symmetric
Channel
source p=0.5

Copyright©BUAA201
Copyright©BUAA201
 1.3 Communication system model

Entropy Claude Shannon (1916-2001)

Shannon argued that random processes such as

music and speech have an irreducible complexity
below which the signal cannot be compressed.
This he named the entropy.
Copyright©BUAA201
Copyright©BUAA201
 1.3 Communication system model

Channel capacity
 In the early 1940s, it was thought that
increasing the transmission rate of information
over a communication channel increased the
probability of error.
 Shannon surprised the communication theory
community by proving that this was not true as
long as the communication rate was below channel
capacity.

Copyright©BUAA201
Copyright©BUAA201
 1.3 Communication system model
Information Theory answers:
 What is the bound on data compression
—The entropy rate H
 What is the limit on transmission rate
—The channel capacity C

http://en.wikipedia.org/
Copyright©BUAA201
Copyright©BUAA201
Comparison of Transmission Property
Before and After Huffman Coding
BSC
p=0.01

Huffman BSC
Decoder
Coder p=0.01

BSC
p=0.1

Huffman BSC
Decoder
Coder p=0.1
Copyright©BUAA201
Copyright©BUAA201
Demonstration of (M,n) Channel Coding

BSC
p=0.01

(2,3) Channel BSC

Decoder
Coder p=0.01

BSC
p=0.1

(2,3) Channel BSC

Decoder
Coder p=0.1

Copyright©BUAA201
Copyright©BUAA201
 1.4 Information theory applications
Information Theory intersects:
 Physics (statistical mechanics)
 Mathematics (probability theory)
 Electrical engineering (communication theory)
 Computer science (algorithmic complexity)
Neurobiology

Understanding of Invention of Voyage missions Development of

black holes the compact disc the Internet
to deep space Copyright©BUAA201
Copyright©BUAA201
Application

Hypothesis
information

Statistics
testing

Fisher
dynamics
Thermo-
Physics

AEP
Information
theory

Inequalities

Mathematics

Figure1.1 The relationship of information theory with Copyright©BUAA201

other fields
Copyright©BUAA201
Thanks

Copyright©BUAA201
Copyright©BUAA201