Department of Electronics and Communication Engineering

National Institute of Technology Karnataka, Surathkal 575 025

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Course Code: CE 830

Course Title: Information Theory and Coding

Prerequisites: Introduction to Probabilit Teaching Department:

Instructor:

A V Narasimhadhan.

Total Lecture Hours:

36

Dept of E&C Engg

Teaching Assistant:
Credits:

(3-0-0) 3

Objectives of the course

Upon completion of the course, students will be able to understand basic principles
underlying data storage and communication, apply basic concepts in information theory
and codes, understand lossy and loss less data compression techniques, capacity of
communication channel, Shannon limits.
Syllabus
(a) Mathematical preliminaries
Random Process: Bernoulli Process, Poisson Process, Markov process, convergence, law
of large numbers, central limit theorem, Gaussian process, Stirling approximation to n!
(b) Introduction
The concept of information in communication theory, block diagram of general information
transmission system, review of applications
(c) Sources and entropy
Information source, entropy, joint and conditional entropy, entropy as a measure in
uncertainty, memory-less sources and likely sequences
(d) Source coding
Unique decidability, prefix condition, instantaneous codes, Kraft inequality, Shannon-Fano
coding, Optimum binary coding, Huffman codes, arithmetic coding
(e) Channel and mutual information
Information channel, binary symmetric channel, mutual information, conditional mutual
information, discrete memory-less channel, channel matrix, channel types, equivocation,
rate of transfer of information, channel capacity, noisy channel theorem
(f) Other topics
Kolmogorov complexity, rate distortion function and its properties, source coding with
fidelity criterion, applications of information theory

References

Thomas M Cover & Joy A Thomas, Elements of Information Theory, John Wiley,1991
R.G.Gallagher, Information Theory and Reliable Communication, Addison Wesley, 1987.
A.J.Viterbi & J.K. Omura, Principles of Digital Communications and Coding, McGraw
Hill, 1979.
Bertsekas., Dimirti, and John Tsitsiklis, introduction to probability, 2 nd Edition.

Evaluation Plan
Quizzes
Mid Exam
Final Exam

25 %
25 %
50 %

Course Plan Prepared By

Approved By

A V Narasimhadhan.
Asst. Prof., Dept of E&C Engg
Surathkal
NIT Karnataka, Surathkal

Head, Dept of E&C Engg

NIT Karnataka,