UNIT 1 INFORMATIO N THEORY

1.1 INFORMATION Introduction to Information Theory Information theory deals with the mathematical modeling and analysis of communication system rather than with physical sources and physical channels. Information theory was initiated by an American Electrical Engineer, Claude E. Shannon, whose idea appeared in the article, The Mathematical Theory of Communication. In its broadest sense, information includes the content of any of the standard communication media such as telegraphy, telephony, radio or television and the signals of electronic computers, and other data processing devices. Information theory has a tremendous impact on the fields of communications and controls. The principal problem in communication systems is the transmission of information in the form of messages (or) data from Source (5) to Receiver (R). The information source can be either a person or a machine. The method of transmission can be done by means of electrical signals. These signals are transmitted via a channel as shown in Figure 1.1.

{x} denotes the set of symbols {S0, S1, ... Sv} sent by the source. Due to channel limitations and noise, the alphabet {x} is corrupted by noise. The noisy version of {x} is received at the information receiver. Two important factors in communication systems are: 1. Distorting effects of channel imperfections and noise should be minimized. 2. The number of messages sent over the channel in a given time (transmission rate) should be maximized. Increasing the message transmission rate increases the distortion or error. Messages that are transmitted over a channel at an increasing rate will give less error. Therefore, the basic communication system can be modified by sending message set through a suitable encoder E to produce a new message set {A) that is passed through the channel.

The detailed functional blocks of the transmitter and the receiver of a Digital Communication System is represented in Figure1.2 and the descriptions are as follows. (a) Source encoder - decoder (b) Channel encoder - decoder (c) Modulator demodulator The source encoder transforms the source output into a sequence of binary digits. It also removes redundant information from the message signal, and is responsible for the efficient use of the channel. The resulting sequence of symbols is called the source codeword. The source codeword which is sent as data stream, is processed next by the channel encoder, which produces a new sequence of symbols Channel codeword is longer than the source codeword because the redundancy bits are built into this channel codeword. Discrete symbols are not suitable for a transmission over a physical channel. The modulator represents each symbol of the channel codeword by a corresponding analog symbol, appropriately selected from a finite set of possible analog symbols. The sequence of analog symbols produced by the modulator is a waveform, which is suitable for transmission over the channel. Typical transmission channels include telephone lines, high frequency radio links, microwave links, satellite links, and so on. On a telephone line, the disturbance may come from switching impulse noise, thermal noise, cross talk from other lines or lightning. The demodulator processes each received waveform and produces an output. The sequence of demodulator-outputs corresponding to the encoded sequence is called Received Sequence. The channel decoder transforms the received sequence into a binary sequence called the Estimated Sequence. The decoding strategy is based on the rules of channel encoding and the noise characteristics of the channel. The source decoder transforms the estimated sequence into an estimate of the source output and delivers this estimate to the destination when the source is continuous this involves digital to analog conversion In a well-designed system, the estimate will be a faithful reproduction of the source output except when the channel is very noisy.

1.2 UNCERTAINTY, SURPIRISE AND INFORMATION

The output emitted by a discrete source during every unit of time (signaling interval), is modeled as a discrete random variable 5, which takes on symbols from a fixed finite alphabet.

The event S = Sk describes the emission of symbols Sk by the source with probability Pk. Before the event S = Sk occurs, there is an amount of uncertainty. After the occurrence of event S = Sk, there is a gain in the amount of Information, when the event S = Sk occurs, there is an amount of surprise.

SELF-INFORMATION Communication system involves the messages S0, S1, S2. ... with probabilities of occurrence P0, P1, P2 .... The transmitter sends the message Sk with probability k conveying an amount of information Ik. Ik= - log2 Pk bits In otherwords, self-information is the logarithmic function of reciprocal of probability Pk of the event S=Sk.

ENTROPY Entropy is a measure of the average information content per source symbol. Its unit is bits/symbol.

E.g. Entropy of Binary Memory less Source The binary memory less source emits symbols 0 with probability P0 and symbol I with probability P1 = 1 P0. The successive symbols emitted by the source are statistically independent. Entropy of such a source is,

The plot of the function H(Pk) in terms of is shown in Figure.

It shows that (i) H is non-negative. (ii) H is zero only for P=0 or 1 (iii) H is maximum at P = 1 / 2. (iv) The maximum is not very sharp. Properties of Entropy

4. Continuous property The entropy H(X) is continuous in the interval 0 <= Pk <= 1. It is depicted in Figure 1.4. 5. Extension Property

The number of distinct symbols emitted by the original source is k and it lies in the X= {S 0,S1, S2,.. Sk}. The extended source may generate a source alphabet Xn which has kn distinct blocks. Hence, the probability of a source symbol in Xn is equal to the product of probabilities of the n source symbols in X, then the entropy of the extended source is equal to n times the entropy of the original source. Information rate It is defined as the number of symbols k emitted by the source per second. R = KH bits/second Where k = number of symbols emitted per second. TYPES OF ENTROPY 1. Source Entropy H(X) H(X) is average information per character at the source is the Source entropy. It gives probabilistic nature of transmission port. The source entropy is given as

2. Destination Entropy H(Y) H(Y) is average information per character at the destination or the entropy of the receiver. It gives probabilistic nature of reception port. The destination entropy is given as,

3. Joint Entropy H(X,Y) H(X,Y) is average information per pairs of transmitted and received characters or the average uncertainty of the communication system as a whole, and referred as Joint Entropy or System Entropy.

4. Conditional Entropy H(Y/X) H(Y/X) is a measure of information about the receiving port, where it is known that X is transmitted, is known as Conditional Entropy. It gives an indication of the noise or error in the channel. 5. Equivocation H(X/Y) H(X/Y) is a measure of information about the source, where it is known that Y is received. It gives how well one can recover the input content from the output.

MUTUAL INFORMATION H(X) gives average uncertainty about channel input before observing channel output. H(X/Y) gives uncertainty about the channel input after observing the channel output.

The difference between the marginal entropy and the conditional entropy is called Mutual Information or transfer of the channel.

Properties of Mutual Information Mutual Information I(X;Y) represents the amount of uncertainty remaining about the channel input after the channel output has been observed. The mutual information 1(X;Y) has the following important properties: 1. The mutual information of a channel is symmetric, that is I(X; Y) = I(Y; X) Where I(X;Y) is a measure of the uncertainty about the channel input that is resolved by observing the channel output and I(Y;X) is a measure of uncertainty about the channel output that is resolved by sending the channel input.

From Bayes rule for conditional probabilities

Hence, substituting equation (1.37) into equation (1.36) and interchanging the order of summation, we get

Hence proved. 2. The mutual information is always nonnegative. I(X;Y) >= 0

3. The mutual information of a channel is related to the joint entropy of the channel input and channel output by

DISCRETE MEMORYLESS Discrete memoryless channel is a statistical model with an input X and an output Y, that is a noisy version of X both X and Yare random variables. The channel is said to be discrete when both X and Y have finite sizes. It is said to be memoryless where the current output symbol depends only on the current input symbol and not any of the previous ones.

The input alphabet X and output alphabet Y need not have the same size. For example, in channel coding the size k of the output a1phabet Y may be larger than the size of X then k > = j. On the other hand, if channel emits the same symbol when either one of two symbols is sent, then k <= j. The various transition probability input of the channel in the form of matrix.

In the j x k matrix, P is called the Channel Matrix or Transition Matrix. Each row of the matrix corresponds to a fixed channel input, whereas each column of the matrix corresponds to a fixed channel output.

The fundamental property of P is that sum of elements along any row of the matrix is always equal to one.

The input given to the channel is based on the probability distribution.

The joint probability distribution of the random variables x and y is given by,

The marginal probability distribution of the output random variable Y is given as

The probabilities P(xj) for j = 1,2, ..., J are known as the priori probabilities of the various input symbols. This equation states that the probabilities of various output symbols can be calculated when transition matrix and priori probabilities are known. Binary Symmetric Channel (BSC) It is a discrete memoryless channel, which has two input symbols (x1 = 0; x2 = 1) and two output symbols (y1= 0; y2 = 1). The channel is symmetric because the probabilities of receiving a 1 if a 0 is sent is the same as the probability of receiving a 0 if a 1 is sent. This conditional probability of error is denoted by P. The transition probability diagram of a binary symmetric channel is shown in Figure.

Channel capacity of BSC : Channel capacity (C) of a discrete memoryless channel is defined as the maximum mutual information I(X; Y) in any single use of the channel (i.e. signaling interval). C = max I(X;Y)bits/channel

Other Channels
1. Lossless Channel : Channel capacity is equal to the source entropy when it is maximum.

where M = number of symbols emitted by the source.

2. Deterministic channel : Channel capcity is eaual to the destination entropy when it is

maximum.

Where N = Number of symbols received by the receiver. 3. Noiseless Channel : The channel capacity of a noiseless channel is given as,

In this example, the rows of the channel matrix are identical, except for permutations, and similarly for columns.

where H is the entropy of the symmetric channel. 5. Unsymmetric channel The symmetry of the channel matrix is not present for an unsymmetric binary channel.