Professional Documents
Culture Documents
Introduction To ITC
Introduction To ITC
Information Theory
Father of Digital Communication
Message Encoder
e.g. English symbols e.g. English to 0,1 sequence
Information
Coding
Source
Communication
Channel
Destination Decoding
Information
Coding
Source
Communication
Channel
Destination Decoding
Source Channel
Data
Encoding Encoding
Channel
Source Channel
User
Decoding Decoding
Example: Disk Storage
Channel
Verify
User Unzip
CRC
In terms of Information Theory
Terminology
Source
Zip = Encoding Data Compression
= Source
Unzip Data Decompression
Decoding
Verify Channel
CRC = Decoding Error Correction
Example: VCD and DVD
MPEG RS
Movie
Encoder Encoding
CD/DVD
MPEG RS
TV
Decoder Decoding
Speech CC
Encoding Encoding
Channel
Speech CC
Decoding Decoding
GSM/CDMA
CC stands for Convolutional Code.
Shannon Theory
1. Measurement of Information
2. Source Coding Theory
3. Channel Coding Theory
Measurement of Information
= ? bits
= ? bits
All events are probabilistic!
Using Probability Theory, Shannon
showed that there is only one way to
measure information in terms of
number of bits:
H ( X ) p( x ) log 2 p( x )
x
6 6
H p(i ) log 2 p (i ) p(i ) log 2 p(i )
i 1 i 1
6
1 1
log 2 log 2 6 2.585 bits
i 1 6 6
Shannons First
Source Coding
Theorem
Shannon showed:
To reliably store the
information generated by some
random source X, you need no
more/less than, on the average,
H(X) bits for each outcome.
Meaning:
If I toss a dice 1,000,000 times and
record values from each trial
1,3,4,6,2,5,2,4,5,2,4,5,6,1,.
In principle, I need 3 bits for storing each
outcome as 3 bits covers 1-8. So I need
3,000,000 bits for storing the information.
Using ASCII representation, computer
needs 8 bits=1 byte for storing each
outcome
The resulting file has size 8,000,000 bits
But Shannon said:
You only need 2.585 bits for storing each
outcome.
So, the file can be compressed to yield
size
2.585x1,000,000=2,585,000 bits
Optimal Compression Ratio is:
2,585,000
0.3231 32.31%
8,000,000
Lets Do Some Test!
File Size Compression
Ratio
No 8,000,000 100%
Compression bits
Shannon 2,585,000 32.31%
bits
Winzip 2,930,736 36.63%
bits
WinRAR 2,859,336 35.74%
bits
The Winner is
8Mbps
2 Mbps
X4
Alice Bob
p=0.00390625
Shannon challenged:
p=0.25
Alice Bob
?
Alice 8 x (1-p)
=6 Mbps p=0 Bob
Unfortunately
Data Audio
Compression MP3 Compression
Communicates
Faster than