ONIT- 4 LNFORMATION THEORY IN TRoDucTION Measure of INFoRNATION ANERAGE INFORHATION CoNIENT OF SyWBoLs IN tong INDEPENDENT SEQUENCES AVERAGE “INFORMATLON CoNTENT oF Syeors IN Long DEPENDENT SEQUENCES Maex-off STATISTIcAL MODEL FOR TINFORNATION SOURCE ENTROPY AND INFORMATION RATE oF Mark -oFfF sourceDr. S BHARGAVI, Professor INFORMATION THEORY & CODING (10EC55) 1. INtRODUCTION Cormmusication imvelves explicitly the beansmtcion A] Injamakion rom ene point to another , trough a Axccusion eee cororount cation Ags q ey ane Transntitin | chanme) ancl Recefver The -funckfenal bloew diagien ao elite! cermnmnusicahin Ayhen ( Injamebfen syhe) ah dre ele Diseate Source |_®_,[Channel]__¢ information sahil encoder encoder] ‘source Electrical communication Analog nar channet Binary electrical stream onal ic ~ 9 Destination }€—"—) decoder [| decoder Sequence of symbols Figure 1.2 Functional blocks of a digital communication system tg be bansmit 4 hie apn UE * The overall purpsse ( sequences Agrbele) eeening tf ae medages yy 4 4 q the source eo Aeetnalion point ak o& sible high orate and aceuragy os f Dept of ECE,SJCITINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor % The Yusctional blocws qo ctigital comune sytem oo {- Information Source + he meaning @ the wad | Injardion Enjewnation the dy is moemage. a be an elechical menmoge 2 intelligence % Wie message may Speech mremege 8 pichure memege 3) music mesos % A Source ublch produces there metioge* te called Injamaken source - a 2, +wo * Tnjamation Sources Can be elanifed be ee : G1 Analog Tnjororkon founces Gi) Diserete njoreakion ounces ae TS bramglamed by the proces x G) An Analeg {inferred en fousce Sado a clincrebe injermation Source a sarptog 4 portry ounces a enfew phone puch as > Analog injnmatior 4 ch #1 el oe emit ome B mle * a Ty camera Seanniey conknuous ampliteal e a Beene | eleclical algrale iat Hme - Dept of ECE,SJCIT :INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor Gi) Dfkerete injamation Bounce fuch at a teletype A the numerical oulpot qa compulen consists 2 ee SG Avserete Sgrobele a lelttens . > Discrete injewation Zources ote chasachniged by he dallening poranetens : a. Somes alphabet Ceysbele m leltixe ) b- Symbol noe Cc . Source alphabet prbabilites Vine A. probabiliche clependence 4 apete - 4 nce. a. FR fource alphabet Bod cole telelype having 26 lebtens 3 xthe English alphabet ple Bim speccal characte te the q aynbels fource alphabet Jor this example consists ie 26 letter + 6 specie) chenachns Ceo _ Fatt stop eke b. The aye! noake ayes ee hich athe teletype producer chanachors Fev example if the Felelype eperatec shen the ab the Apeed “4 to charactas [eee : Synb! aot [% lo greet [se : Dept of ECE,SJCIT 5INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor y teletype vs producing meager fa English | cthen Beme lettane appess mae feqpenty shan Obhens Fa Example the leer E' appears ale "These Shuchmal propabes eee Oy endividual fyebola often than eles “2 4 Ayrnbol Aegpences can probability q Cetuhunee Ff ard by sthe emelitonal properties C probabilities) Fo ccounee a From the dfigue, lee as autume Atacrebe source emitting % athak the {yJamalion ounce es discrete mevage Ayrobel « si, se, oo Sy prbabilibres A oceuemer given by py Pa, oo Py areagectively - The fur gq all these probabilitie vnust be spel to 2 : 1 PitPa tes tooo Tt Pa con fo. | Dept of ECE,SJCIT 3INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor 2: Source ENcopeR be a tnpuk to the founce encoder eo Bounce alphabet aleing 4 source a. uo whe p80 5 cecuntng ak a nake 4 tae dyrabels [ree The founce encodes converte the ayobel Acapience Toto binary Aeapence 9 os 4s by alee ele end ye ayrbel - Binary code because the hig! otrery 4 bransmition - Tame ane tee Mg mY G2 Fined legth binary coding th binany coding codes : GI) variable leng ined The pimplest way gq coding ao oer & tee i tn the engi binany code wad jy each ayes! ; Sopa Aegprenee « This coding is cyprdeot on % 7 bilikee fo individual Aprbole occur with eapel probabi a platiekiend Indipendent sequence - 7 2 Wi In mest prackeal gituakene the ape ee uneqpel probebiliter. The source encoder then 5 Netable length code -utcle do these Byabele - ace encoder aon block aige, length 9 code wade | awenege ate a Ly iy she he impiitant poranetens qx 4e Dept of ECE,SJCITINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor 3B. TRANSMITTER MB Tranemiltes couples oe tnpuk wemage signal oe the channe) > Sometimes i may be posible couple input Wransduces clinectty to the channel - the ecenany Fa efferent branemiasion Tb Ms opten ee we prcen and wodity the iopuk ag He b, Sigrel preetig opmaltme peyomed CO oe Nennarnttear Snclide amplikcehmn — dillening CHANNEL ENcopeR 4 DECODER poubuahiniend pueda TO channel coding is prockeal meted 4 sealeing hegh leansrniaion nreliability and eyrdengy - channel encoder are The inapartant parameters q+ she method q coding . anoke fH epengy q tthe coda , ena conbed copabilibves ) coraplen ty 4 the encodan - 7 The channel deceden secovers the ie ; a bearing bike Yirm He coded bivery * Ent debeckon and posible cduchon 4 alto Peyamed by he Channe) decoder - The cleeoder epeotee either in a blecw mode ae a earlinuoas Acquenbol mode ependiog on type +f ing wsed in Ayttyen . The conpplenity a ~He ecod Dept of ECE,SJCIT : =INFORMATION THEORY & CODING (10EC55) Dept of ECE,SJCIT Dr. S BHARGAVI, Professor 8. CHANNEL tHe elechical OC goes ae destination - A communication channel prvides Th channel my be a pain 4 pines 21 free space % Due to phytice Linnttekions Carernunication channe} have only nite banduidis anal the infection beaning aiqral puppers amplthade and phase ee alee Bigqrel pone decreases . nk > Faunthemne the aig is casuphed unuanted unpredichab le Aigral called noise . The effect ae Te te Convert fore gq the OS into Ns and some ford 4 be 4] the 1s wo os. THE aignel ig then caaupted YY noise - 4 Denepersles a oe eee un ah : ont and comente ik to an cleckical Howejnre Aattatls oe channel - 66> Mopvtator eee Ja Kanemistion oven the cmenmunicak ea do mabeh she { nee oie! billy ce a a Modulabn can be etek dhe appete gq channel notre | Apechun the hrana mitted a chorakurkes , t prowl the Cope rang Kignels and bh ovencome Aome eels limidahione -INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor % The exkachen 4 sthe memoge jer she injsmation beaning wave}fion produced by the la ie caccomplithed by the Cte | % Fa a gi mnroduladien the mest ienpartank Oa 4 : ; parameten the demedulabn tx He ed Aemedulahon . 7. DecoperR 4 DESTINATION Eee cae The ounce decoder consents the binary oukpuk: q the channel clecod i lences aye Sequence + os Variable length codenads is Very complex - % The funchon 4, the Aecives (peslinahon) '4 ch ik to idenky the sgrbet feypence and makeh pith the cdaect Aeqmence Dept of ECE,SJCIT 8INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor Explanation OF Glock DIAGRAM oF AN Information SYSTEM Fan Communication the basic ‘npul should be inferoation Bounce | Which can be analog eh ene We Consider it as cfinerebe infermerk' en Aourte . ae Inffmeken ui be in whe drm 4 syrnbol and 1% pamed tr the source encoder - Source Encoder Will compes the Afge 4 Infamaation Code ee ne aybele inte binany | bibs channel encoder . Th the channel encoder the vise moy ae added. BE He noite get Do ba it Cee Beapence and hen pawed he the called as cheeneume - ene |e opel te eantdee dhe data jor Aounes encoder +o modulate and the ofp 4 channel encaden ill be ia blroyy sagen « A. H, f ja weation Modulalr on fe < proces 9 erying the fe q his Aignal pear Councet Bigned and the olp fs elechraol migra) Pest qe Ay the demoduladr ee clarnel : ex tHe Demodulehon [6 he preen 9 geting 5 binary Bepence fiom Me lechical signals ancl is gre te channel decoder Dept of ECE,SJCIT .INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor ee deteching 7 cAtwcking the anane |motee . Ot dunes place fo 3 skips: i) Ewa delechon Cit? pests bi kty 4 era cimckom — and Ewe caechion - Now the oukpuk Bile be Ge ea dam q binasy } a and i peuteal ao Whe Zource Aecoden . Source decoder % used +o convent the injawaken uhich fe in He dar 4 binary | bik fob the aiginel Aigvel te Aarobele . wtis Biqnel 4 poucd to the deslinaten * Pa hieh Aeceiver £0 a he ingrmah’on vo Kent a Qeaved bac: Dept of ECE,SJCIT 0INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor 2- Measore of INFORMATION UE Get re eee % nade to Know ana Compare the Ingermahron an content’ ] vanious metag es prduced by informal on Bounce a measure 1% aptentitahively mecetany fo knee tak infnmation content . Bourta is The culpa ga ciserete injamatron A metoge dhak consis 9 + Aeqperee 4 Aymbele The Ayrnbels eed an 7 2. Bounce oukpul ' fee fee . wedeled as « cliserete sander a 7 which daxes 27 ayrbels pon a doe dinite alphabet Se: Co send Was . ee prbabilekes p(s> Se) = he This zeb g probebilthres woust Aatitdy dtkten ket . >in an Keo b ele le cubke “mes 7 i phich he eslabliehed hed tHe mathematical al head J Communication , bosie theachkcal bound« da Sie ponents 9 Comrnunt cation syctime i Dept of ECE,SJCIT 7INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor Shannen’ sheay fe based a prbabilistre models and commuricalven channels. in Infermeadion Bouncer “Shut on an Gnlutkive basic the amount ef Inlnmation Received porn he knowledge 4h fn ee) ow ee he avent-” ecewtence og an event Pro babi lity 9 cceumence 4 contain mete infamalion than other Some menage menager - The menage axsveiated Hit an event least hnely ee inprrnation - eee a pith + oceun Contains Suppte ar infarmedion ounce Powible menoges enema BY prbabiliker 4 ocemnence Pi, Pa Po. ea Pit Pat Pat --- + Pye! edule athe infer ion cortent 7 ye th menage nrelate d Acending +e owt B the amount 4 in}emal en in dencted by (oud, muse ie oes > Also, T Come) must approach Furtermae , the infra on content LC a Negative since each meweg® contains BOME ingen Ae hast | TL Cm) oy i aqpal 4 Zero inven el approach L oes ie nen J must be Dept of ECE,SJCIT -INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor Summaniging there megpinemente Lene) must balighy : % Lex probable an event is the mare raferenadion We rt when Tk eccuns ee ee © % Tp we ane absolutely contain q The oukcome 4 a7 event even befate Tt occurs thane fs no infamakion guned =) — @ er Ge ee ee eee pevde beings abou! % the occumence J a> Afome BA 70 Entermale en bub never a lo gq Infnmatton => Tern) =? 4) — © A ol ae gtatisheally independent 2 x (mem) ¥ Ih mM. and my ote > Zr and my) Se ae © athe * . Conhinues Juncker ae th conshainks aired in eqpertont Ape Vos Aepned loqasttrrie jurchen an measure J ingermateen Dept of ECE,SJCIT 43INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor * The base fn the logan ttre {n eqpakion ws) clekeamines the unit auigned to she iqbamedion Content . —> Th the natural legen’ tren le) hase is used , ee ait te mae Th the base is to Hen whe anik Te Hantley nN Decit . a . den the anit 4 Ty the bare fe > infomahion ee ; an abbreviation {2 binasy digit . leg exprttion Te chowten Jer measuring infarnahion? A Logarithmic enprettion i4 chooten Jar measuring injermatron became 4 whe detloning mearone - The infomation contenk 21 Bel] - infermation 4 = ode €annot be negabre . Each ee mus, conkain centatn amount 4 Ingrmation « a | The louesl pascible self infevwation * zano” which cceuns Jn a fue event Arnce pel Mae iqjermadion fs cassied by a lens Trcely ae Ulhen independent Aymbo le So bepemitted re total Bel - infamadion must be opel to the Zum 4 individual selp-injermation - Dept of ECE,SJCIT 14INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor Examples 5 pasible messager dust, 4. A Source ou me 4 a Q each menage inkaval . The probabilities aq these metsages ante sh mela P32 Ve Par he, Pe | Find the injamadion content q each aes Solution i -tog,(%) 2 bit Tom) = lege =~ haa Om lc loan Pea tng, ('/%) 2 bits bibx Itm3) = - log, Ps = leg, (Ye) = 3 bike Tm) - 10g, Pa = - lesa (ue) ee . A bits seme) = -tea.Ps = 7 tae (4) terete merotiylen ounce (Source c), Ce ca A and 8), each Lineny Sours Consider that oukpuke shoo bihany ounces C Soux Aounces Con buting tome bit . The two Cndupendent . Bhak ts within the fowiece © are : the faftimation content q exch oukpuk fore “O° fouree “c' 2 Setzer - ek Sousce Cn bibubing ene bit oof of doo bite they pill be eqpe probable pea) = pls) = 4 = oF Dept of ECE,SJCIT 15INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor > There ane 4 posible oukcermes Looe to" each nih, prbabily Per = pcan Pus) = Cos) Co-s) = 0-25 Since the Sources arte Independent . The in}imeck?on content gq each outpuk Bip gouyee al 16 rte — leg, PLE) = = leg Co.2s) Sele binary pen Se occun With probability ole: shen Ya and ‘1 eccun with probability 3)a caleulake amount egotrontion conveged by each unit . Solution plo) = V4 pu = 3la 2 bits lo) = lege * plo) i ae ee J Independent , likely 27 ‘indepe A Th there ane M eqpally Keely os wetages theo prove what amount 9 canted by onch mevioge wih be T= 7 bite | Where Hs 2% 4 Nie an Inkeger* Dept of ECE,SJCIT c bite UL 16INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor SoLutt0n : \ hy Well od Since all she N ae one eqpally ly independ ent i Probability 4 eceuunce 4 each menage pill be BiCM) ey Ie = leg, (4) Te = loam = tog 2" = Hence proved - oe tarcnetns ranted by memoge m, 4 2, i6 the Fnpornal en comed by menege me amount 4 inpareaion casmced shen prove that the Coo et Coraparitely ee ed i Sotution Wek T Dept of ECE,SJCIT 7INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor AVERAGE syntors In Long “INDEPENDENT SE UENCE, icons pots TBE pe ec, NFORMNATLON CoNnTENT OF se ‘S duced by Fafnmredion Source eonaist syrobole - The eceiven | 4 Doe, Memages pe 4 Cea ence 4 fnkerested mainly Gm wads phrases 27 Lhaees the Commounication Aystem hes 4e deol il, Individual letters m1 Ayrbele - “The mehic de measuning the fnjarmaction fo be branamitted Has developed by shannem , wthe Auch a mebic WH called the Enhory 4 ened: Aource - contenk ZERo- MEMORY SOURCE i injarmation > dE aepetents a model | * Atacrebe Inger Bounce emitting a bea ence agrobel a a 7 dined dintte founce alphabel $= § $1, $2 $s,7 do some kxed and one Stahkalvenlly independent a called Symbols ane Selected aandoml probability one another - Such +yre 4 Bounces ane * merntyles ” a1 pounce’ - Zeao - mem y oe any too Alle and “There fe no connection he nee eee ee ade Called “ memdyleu” 8 “ Zero-memay” fources « Dept of ECE,SJCIT 18INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor ae Lek us consider a Zeno-memfty source that emits . ore af H posible symbole 51,52 a a skakistally indipencent Aeqnence he cthe pobability abot duning G oceunence porteslan Ay 4 Ayrba fokewel dee oot depend ™ the Ayrnbets emitted y He ounce using dhe preceding Aymeel fabeavals . + eb fe fe be the probabilves +f Occumence 4 the OM ayrbe aespeckvely * L the ini en bel alow, In mle mesege coating AY Fo she average ' : aka the Ayrobel 5, will cco RN Aimee, r win cee Pr times - the Aynbe 3 _ + Treating Individual Ayrabele as matages 4 aN ene, the infamaken content gq he ae can be datined as log, ‘Vp: bits: Se bakes one — Vence he P;N occuuncet Caw inamabion content PIN tog, toh. The betel injnmabion conkenk g the menage oe q the Conbibulten due to cach g He tt agenbele 4 the fours alphabet and te qe by Titel = z we; bea, (4) bE Dept of ECE,SJCIT 19INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor grnbel can be oblained % The Average Tnyernation pe S by dividing he total Infewoalion content 9 a meage by the number 4 Ayrobele fn the menage 5 Os w 3 pif ba (a) pits | eyabel Totes [| > © aa the stanky mathemakel bite eyo > Eqpaker @ sas uted by sharren pork fn hie eugene! presentation gq the they 9 Communi cakon « wobol % * The Average Injewnakion Content a sf w nb : alo called the source Enbopy Some Properties OF ENTROPY ff hes. ee zene ee e wnattemalieal model is defined by espatone O + © below : See sa P(sesx) = S* we o,', . fe pcsssi) = ® we! hae & Pe =! weo The Enbopy Hts) 4 pach a ounce o4 Hts) 4 (og, 4 Dept of ECE,SJCIT a bounded os dellous: isINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor 1. H¢s?20, i and only ye the probabilily Pe =! dh some K, and dhe semaining probabilities in the et ane all zero. This lower bound on Entepy Cbrusponde +o no uncertainity : H(s) = tog nm , it and only i Pu Ye daa Cie all Syrabele fn the alphabet s are epiprbctle) . This upper bound en Enbopy cdaesperls to vith ao alphabet Ce % In genral, ja a Source . : is attained her ” symbole He manimun enbory ie a enbet probabilities ane aye! ] the & y IK given ee Ce Weg 4 uhen P= Pa on- Bmax = '9," bits | symbel Source InFoRMATLoN RATE R ane ee ee rae —o : Sous * TL the fymbole aste emitted by a oe : ( bole [see - The overegy dived time sake Say Ms ay as 8 na’ ta bitefae aan be A fourte Ininmation stake Ro in > " ' ies content ag the prduck qG the avemg® ingore per syeobel and the Ayrbel rote As” | Roz ash bikkee os Whae ag 2 syrobole cee Lets |tqanbel ge [sec 2 Aca 21 Dept of ECE,SJCITINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor ProBLemMs : ole 1. A discvele Source emite one gf ft Aye ence robot probabilixes ane every millisecond . The Ay Final the Source bh va, Va Yu t he aupectvely Enbery and Injawoation mate . Sovotion e J Enbopy Boe ee ea ee del 2 ce les 2 + leg, re) > CA ae ee Ge + 4, beg, G2) 16 ea) ave apa et log 2) + 7 log U : A leg (ie) + L tog Le) \e » tog, (8) + ra 2 = Ge 4 os 0 Sie oes ao. "2 iets bite] ayebel wn njrmation sake R= a. * caeetaee: a eo aces: bite [ce Do aoace theds emit ome shabebeally independent Ya and Ya 2+ Find the enbopy 4 3 Ayenbol< Ae Cpdee an Bepenes with prbatilikes ee Dept of ECE,SJCIT =INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor Solution Enhopy He Lt A tog, (4) +L og, (4, Steg Gay +1 19, 4) + be9,(4) 65 405 +05 B- A dicch somes ewh me q Aim Aymbelt ane once evany msec + The Aymbe! prbeliisees "| va Ya, Ye Yu tan + Toe meeperhrse : a the source enbepy andl Tnjamabien Me™ SoLuTIon Enbopy oH = as ro1s6as ce pits HORS * ONE a7 os FOS TO ee sae wits | grob 4 Tipteers qs1s) E(iseo) #2 Dept of ECE,SJCIT 23INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor A. Consider a diserek memirylen Aounce with source alphabet s+ { 50, 51524 and oumee Atabithes Jot os ots}, Coleulaki the Enbopy 4, the Source » Sovutton Eahopy Wo = Py log, bite | eqnbel wig log 2 = 0-7 log, cA + O18 tog, + #0 1s log 5 o-S6on + O-elOS + OAS Lee Le eta bis [eqrabed bits . 5+ Find nelakinship between Hantley nats and bike Soluiton Wet I 7 + Heatley —-@© \ ke = © =a oe re leg, bike 3 © aw Yantley _ mate eledvorchip From eg Ca) * 1 Hantley SS e990 F \ loge i lea 1oF \ Hanttey = 1e9,'° nak 1 Haatley tor) = 2-303 nade Dept of ECE,SJCIT 24INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor (ii) Haatley = bike aelatienship Fron ep%ay Te log 6 Hontley From ey” (2) \ Mantle = \ Haattley oe bike tng > = lng boils or) . 33> bike 1 Maakl 4 = Dept of ECE,SJCIT 25INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor 6. The output 4 an ingnmation fource consists 1So eyrabole aa 4 heh cceun With ow probability ‘Joa and the emaining sith a probability imbole [see - 4 Vase the ounce emile 2000 SY i leat! Arsueing that the symbole ate chaen independ 4 find he svenage injevnadion aoke 4 wthig Bounce - Solution WKT Average infarakion Se ee ee (130-32) : ofl 6a ey ast | x ng we [ay wast dase + [dy May : 3.aas = 0-093718 32% 0.03340 *NE = 3F Hos 6-9413 bite | eyrabel Given ms 7 2000 syrabte | tee R za.*h 2000 * 6.9418 [es aera bits [see : ae 1. the inkimalional enfcc code uses o *0qen’s ov iw ard dashes tp Wanewit lelees english alphabe - the doth it auprarente d by cusent Pp ' alunahon a ion oe duration g 3 un an G \waik - The probability G secuence 4 ls is Ys . the probability 4 oceumence J * . eo ee Dept of ECE,SJCIT xeINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor (a) coleulate the Infomation content gq a dot + dash. (b) Calculate the average Infomation fn the dot - dash code - (e) Ansume thak the dot laste tmsce thich te the Lame kme inkiaval as the paste between symbole . Reel dhe avenage noe iafarmaation fransmixtion , qoutes clo and cashee ane present > Since onl Pdot + Pelash Pda = Lf Plek 2 Pdok * 3 Peach > that Pao = ! Pdok = Pdesh = log & log - B log A + aos rae a ae a = 0.327% +0 Ww = o.eaTs bits | Syabet Tdok = imee 5 Tdash = Smfee | ote dosh Spaces te 4 ximice) = lo meee oe (Geel mee tt ee + Aon symbole Jsee Ne = A Symbols |r nebee Inlamohion make R= as #H w Aco € 0-827 = gages bible 27 Dept of ECE,SJCITINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor B. Consider a telegraph founce having 2 agrnbele alee and dash. The dot durakion { oatec and the eee ee q we aot Auaakion . The Prbabiltly q He ek occuring ‘a baler Kok 4 dlath , and ine beFucen fymbole ib ovate. coleulake he infomation mabe q the telegraph , broree Producing ee ashing q 1200 syrobole - Sotutzon Lek the probably q clash be ao probability 9 dot be Cp enatp hie dospety pets Pdesh eb, Pde 2 ES) 25 Te coleulate Enbopy (») tog (3h) -08 28 40-4206 cot = He thes + % = o.ares bi | onbel alee Xo cobeulake symbol mote LA) Sl) 6 dete 3 (Tdot) 7 Time bebocen spnbele oo as ‘g(o2s) 20-68 “iT 2 Tek * Tdesh + Lice + Teyeebel ) Number q doe = 2 l200 = B°° > Numba ¢ dosh = tu woe * o° * " T2 (00 + 0.2] + [400 ¥0-6] + [00 > loot 240+240 2 b640eC Dept of ECE,SJCIT ord 28INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor OP eae sybole fee e+g J. Tajamation make R= ne #4 1e1s # ONES Ro = [.721)8126 bits [see A. A black and white Tv pichne consith q Sa* lines %] picks infanmadion . Ansurning cach line consists 525 pichie elementz having 256 brightnew level« od che pickne ane nepeatid ok the ube q selice - . Caleslote He average iqhtmatien nate emveyed by a Ty sek to an Atceeven SOLUTION One pichne Contains sas bine i lements Each line contishe S25 picture elem pichwes ane pert of a nate 4 0] see Picluse element make Ca] = $2 prchae Skin xX 525 ne | prehue % 5° picks | tec = G.26 278 nie pichue zlemenk, fic oe = leg MN 8 : = log 28t = lege = € bits pichee ibeak asa bis Josie as 108 prekoe elemenh lee sd g- 26% 66. 15 Hbps Dept of ECE,SJCIT =INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor lo. A cand it dawn from a deck 9 playing conde. (2) You ane fajrmed that the cand you draw 14 o Spade _ how much inferrnetion did t” Meceive - Cb) How much infamation de yee eae ee told tak the cond thaé re drew fe an ace? (Ce) How much infermad'en do ian Secive i Yor Ae bod that the cand You dnese fo an ace 9 spade? (A) Te the indarmation obtained tn Ce) te the Bum 4 information content 4] the mesages “ spades” and ace Solution tn a dees 4 52 (a) Since there are Papede 2 leg 2 obi Sealy infamaten a Tspade = Cand s Go) See ge ae ae Pace = 4 . : ; = leg Sei inferakon Lace = “9 po CV Since here th oly one ace 4 ~ spades fn a decm Ae cde Pose § - Spade ae Sel} infimedion 1 of- sped = Dept of ECE,SJCIT oyINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor (A) Yee, the injaimation obtained in Ce) t4 dhe Bure 9 injamah'on content gq the mesog et Spades el aes Since downing & ‘apedes cand and deawing an dee cand ane hoe independent events and hotel fel) information mast be eypal individual Sel} ingewnahion : as + Tepade L ace ate - 9 ~ Gede | LE = as and is We An analog Le is band limited oe “had Aaropled ak * ny yet Rake "+ The amples ane ye oo into + lewle - the Quantigahen levels a4 abies Pret “#7 Hye ounce be indupendent and eceur with pre a Find the injarmah'on sok. 4 . Baa y bits | yrobet Pes Enhopy wos) = > [5 log a+ [3 ) 5 oS +h OGNS Solution Hs) * Somer Seale 2 (05806 # sa b | tyrabd hte) = te . Ginee the signe is somgled ob Ny : a2*e] sare Soo \ * = org # 4 OSD ls Tnjamatien rake i bithee = 4002 1Giobib [+e a Dept of ECE,SJCIT 34INFORMATION THEORY & CODING (10EC55) Dr. $ BHARGAVI, Professor wh, In. A Source twansmite two independ ent memeg <# pr babilily af = Ger) aapechvely » prove thak the Enbopy is manimum hen beth dhe meneyes arte eqpally likely» plot he enlopy | Aomner ven as Probability (o? x20 MH when pet, the Eebopy vCl . luc imum Va! Te En Ws) attains 1G man + bop i = \ bk wheo P= UI?) et i nat = Aymbole ' and 0 one eqpally probable : Dept of ECE,SJCIT srINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor Proof For LouleR 4 LppER Bonds on Ha) facet + pln 3) 0-09 (25) 2. To gind manimum Ninit dwt. , de ' Ane = A Er tog, (4) +07) 8s (te)J 2 + [- Pls? - UP) tog, CP) og UTP) 7 a Ey deg) use os, P i J -p)d_ beg CP) u ne id, Wut [ leg x = leq X leg eq og, - Aau Die logeh + tog, cr?) - ay . FP tog : (i-p) A led ire) Ap ——— tog > «Le bog, (iP) - r ye (ep) - ae a) = - log p+ tog li-P) c-') Dept of ECE,SJCIT 33INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor Te Find maximum value d wos) 20 ap lo = beg Cir) af 3 2 : Por oP SB pe Ye Thus when pe yy Cieprs yy the HIS) is marine. : wilt be Vence hen evank ane eqiprobable En bepy, maximum - ‘ en 2s) Slog Mt Hs) cannot be ng ative) hence | o an plok gq HCs) Nensus probability r Dept of ECE,SJCIT 3aINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor an indupendent Aespamte 4 o's and 1's with probabil p+ in? respectively pt the enhopy q ~he founce veusur P Solution (eR) A binany source ig emilin Enbeopy 4 binooy Source “6 aa | Hos) = SB fi leg coe vet fia, big oo HGS) : ray + de Pay Pp) Lek peo WO) Fo. beg Lt bgt = © bik |p Lk peo-1 Hos) — + 0-4 vase as abt His) = 0-2 leg - toe leg 2 20° aa bib by pee3 Cs) ely ae 4 ee =o. eet Bh fyen S83 Soe ond log by tort ligt 7 O17 bits [ym pro2 pro-4 HLS) = : ik Pros Hes) = os beg de tostegt: # re lyeo ° a WE) Foe leg +e toaley ty * o-anl ni o ea bil Wis) 2 07 "des god bog dp OBI b : wos) = 0: 8 beg 5 ate lag 2 O-taa bik py L269 bils ky wos) = 0-4 leh tol bey 55 i bits [sy nb- ws) = tly 1 te Cees Dept of ECE,SJCIT 35 i esinntnaeNniniesnissatissttstnntn enINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor ey Or 0 OF OF OF Ob OF OF 13: FR a discrete rmemsylen Bounce there ane 3 Aymbole hth probabilibes PF, 2 and P, 2 Ps ° na aketch Determine the enhepy 4 dhe Aounce he vartalion Ja diferent valuer 9 %° SoLvuii06N > 3 prbabilibes are PP + Ps wet 1 Pr hte * Ge Pots Gg Dept of ECE,SJCIT xINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor + (et) ton. (Z,) = (Db, ( ay + (1-2) leg, (4) oORCE Extenston " of A DIscrete enonyLees so! + St te usedul +e Censicleq blocks rather than ‘individual Byres, WH zach blocks eonsinting qe Bacenss ve Source. Ayo! os ee oe geen ood) a berng podced a An ertended gounce with a Sours alphabe! Lg has Kr distinct blecn there te dhe mraber 4 Aishnek syrntels tn the = ounce alphabet s4 the Digi? fource Dept of ECE,SJCIT 37 bos” dhok-INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor H Tn the cate q 2 Afserete memayles source, he Bowes Symbols ane stocistically “ndepend ent Hence , dhe probability qe Source Bynbel in Ss? ‘a eqpol to the probabilities product q the 9 Source aphele oy oo ee founee th Ce A elt mn times HOS) 14. consider a Aisevete am fource with Source alphabet 5 2 50,54, 52} with probabilhes pr the xh suspeclely find he Enbopy 4 Hak mda source and Beond Ader source HS) - Sowwitiey 3 wis) = P; bog.“ ree AL beg oe bg Fete FS » (os ao 640-5 FE o& HG) = 3 bits 2 24 orden Exkeosion, qe Source cu *% Llih the source “gui s consiating 4 Bayme has She source alphabek s* 9 Oe mma nine Aymabole - Dept of ECE,SJCIT aINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor & Lee the nine 4 Symbole of es er ae | le sle..| Lt sqcbele s Prbability 7 ( +1 a! ye Le tng Co) oe bt, ee (ie) + 09,0 7 2 16 \ cewg, 4) tg, HE 9 (2) +h bs, (ed Li) \6 s > 0.26 ¥08S + 0 oe 2+ jase o 817 F 0-315 40% ace) 2 3 bb ican nee lewet Hcst) 2 na) aais 23 bite - \S- Prove HCs") = n Als) ; i : babilites % Consider the Ayrobee are Gene] with pe pein, 4 i = poe P °9. 7p) we td nde erkogion | Aosace extension. ce [2] *24 numba g syabele gre hy 39 win, bavt Dept of ECE,SJCITINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor ss, oceuning uit, probabilities Pt, 2 Pe PiPa vA acknsi Sum g all Probabilives 2A aides exkasion Bounce 1s (c. : Pre 2B rh = Peleg Stee P Jee! °3> (za) (5, a % bos, ie +e * tog Fe og. te + 9p, 42> e 2 Pte, Le + Ph log, a °°, oe ‘ 1 Dia tai pe leg ee soa a log, ma 2 PPi lea, AG iS Pe 2, teen] leg, Ts oe Cart) 109, >, a Since P\+ PR, =) Ae oe! oe . 1 "gg ton, 25] - 7 Gl aoe Fe [aed= 2 Ws) — © od wit have 2° = 4 ander extingion 0 = 2p, bea, g number 4 3 Srevilealy Dept of ECE,SJCITINFORMATION THEORY & CODING (10EC55) Dr, S BHARGAVI, Professor 5,515) 525.5) $,5,51 $,$,52 55,51) S, 4.52 , 5,5,5. $2522 With pababiltes pl FX Pe , , ° a 3 chive Papo PP PP, Pate CC re eee) : «a | bina “. Sum gall prbabilker g the 4 extinde ys Bounce i+ oo : pra srry tana t Pe = (Ath) : binary sion dhe basic mn genenal mi enten a Ta g ae Aouace will have 2” syrobels nth extended four is giver by PRopert1ES OF ENTROP __ |! ory % Fn a Source alphabet s= §si,S2 --- Sq} aes prbabilives pe LPs faj---,Py J where y= number 4 Aounce Aymbols wea & % The Enbopy duoclien is giveo “24 Hts) = Dept of ECE,SJCIT aINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor listed belo: % The properties q the Enlopy are © We Enbopy fireton i cominuous dh 2¥ey imclepenclen et CoO] ce Ure et deez the evbory cbanclion Vasiable Pye in he continuously fen ob 1, Se @ The Enkopy junekon ie a AY arguments te WI] Pe (ora Lae ee) Pe] , a momehteal qunetion 9 Se Ce the value gq (5) Semaine cthe fame imepeclec eLed, - de sores 4 the locations gq the probebilikes © Tht Sa Sp Se vit prbabithes. ae br Pe bed 3 fer 4 Pa, Pa a Cy bee ee Poe fies ity wy te (Sa) = 4 CSe) = Ce) ® Extrenal property . “the fame Source F wih oy eee wi probabilities Symbol Ses. os a ea fie ta : iven The Enkopy 9 5% O 4 id i 2 He oe eg ee Dept of ECE,SJCIT 5INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor > Leb us now prve shat the Enbepy #65) has an * upper bound” by considering « apart ty [iegy - BOI [4 ue obvious Fen the above pr Nias above the leg Wh te evidest that dhe shraight aaifarnt © CORTE line ge t-! abooys yelnx exapt ak act. The she Bhaight line drone athe “Sangent” po the cuwe ok 221 - Dept of ECE,SJCIT }INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor Pe be gek Hee) ae pS te ae Are . i | © co. as , thee nay Ri EI OE Similaaly binaay Sous ce. My Pi F thea time Z fa ol) cere aa 4 We get 2 Pi kn yp, = 2Pi [!- +] ist fer fe Nie by bat He get ¥ ¥ i lg e SF Py lowe, 2 leg & ee - 2 0 . ia ie ie) wes Fem ey Lins vill be log yO? 4 Ry ‘6 choy. 0 z0 leg ag HS) a nts) = 19,4 — © Te eipakidy Agen hel de re phen Pink =e Sy ay values Q ie = tehed ‘han Lthen the Canditon qf eg?le) i* pocke ged the enhopy Letras “maximum , gerd Dept of ECE,SJCIT aINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor —@ » hea al) = aad bt | menage Arnbel H Cs) m a “The Eobepy aitiins oa” cnaximun value * equiprbable the Bounce symbol « become @ Propesty of APDITIvaty into 0 Sub symbols ne Aplit the Ayrbel #7 ee ean * Seppe afr s uch ot eo Sa Shr ae a ’ ’ Buel occuring that bh Prbabitthe Py, Pye on» oo hae Pa = Py,t Past Pays wry a = Zo pay a Jer : Men, apltted ariel Enkeopy ws fe n 4- a 2 Eri ee 2 fas “2 Pag tet bis) = Ch Pe os ' + Z Pay bg Py jer (fen &709) Dept of ECE,SJCITINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor Hs) = nls) + 4 pavitive pony Free Pag Z Pay fa al a into Sab- tyrale conn ce pemikening 4 Arable eee Source Effiriency in denoted by 1b qe 4 1 2 HLS) 4 (s) maw “The “dour Redundancy i¢ devoted YY Bas St oe a y ew s s a ee ts Nowe Mo x Ra can alse be expr 4. AVERAGE INFORMATION CoNteENT OF SyNBois : —_— IN Lon4 DEPENDENT seas The fources ane Laid to be preducing alatakaly indapendenk Kespencet | iy the ceeutence 7. 3 : pankeulen Ayabel chasing a symbe intuwal _ does n° din the pubabilty q oreumnces 9 ayabels desea amy otlen Aymbel intervals. However, "a od i bole thak ots all prack eal Four. emits Aaponces BAYT Aiadis Kelly dupeadenk Lire Seunees have memsry ) Dept of ECE,SJCIT éINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor The Beqpencer gq leltere numer ale and Apectal charackerg however ane mot completely random Te general they gem Bentinces and have Alotishcal shuchae This Btakslreal dependence 9 Miche ee he eunh g ingatraken Coin) ouk gq Such 2 Aounce compared te amount ee eg ee a 4 ; indeperelent emilting the Same Aet 9 Aymbola in 4 ces. gp ee ee eee plalsded model i endent Aepue dn sources exnitting Apotsle in depend Aeypuenea + ine ee ee ey dagi oo 18) He Eohopy UL) and the injernabion rete For TNFORMATION Source A Atsume Aisected for inthal de He ginal shke «The bran sition a : cgrespenalios probabil and he Aymbele emi Hed ra Bhown mat 40 vaniour bansidient ane usually 6. ENTRopy AND INFoRNATION RATE OF MARK-oFF SOURCE % The Enhopy a te ounce (Hann off rede] Sune.) is difined a1 the teighted average 4 athe enbopy q tHe Aymbols emitted Jrorn each Blake where she enkory hake i ete by Be, depined : he auenage nfemnation content te Aymbols enitted jroo the Uh pom alate Dept of ECE,SJCITINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor * The Enbopy 4 Hy ee se thes ie avengg < 4 te Enhopy q each shke. te AVERAGE “INFoRNATLON RATE a 7 2 R the Source is The average ingemadion ake RY depined or : Recah brs | sec Wh He ie the number | take bansi bi’ ne Ne iA the coat Per Seema a athe Agmbel § ae 4. erailteng % Hence the polabity q te Bounce i iven by use - symbol segpence ae %* 4% 7 oo 5,5! = oe a plaey = PCSZ! 42702 OS eos) 8) p(s 7h $808) Sie te ba aiken He be gk : pane ee ane alisjer 4 AB SS 5223 22 5,2) $374, G PC eee y+P ls ss pcas) = i. Pr PCs A) Dept of ECE,SJCIT = iii i i a a a leINFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor % By using chain aule op probability we ak C4es oe ae p (#2!) p(s323 [2.2 52°!) = p(set) p(422')S12 9) e(s3231e 70) Exanpte > Fa Seqprence loa2532 Zo eee 327 32 Dept of ECE,SJCIT 53INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor Eeanite considan Q Zounce emitting ome 4 8 The probability g occumence $ + Ayrobel depends Parkiculan symbol in ypusen and He symbol immediobely preceding ike The past influence lark only da a dlusedin p —™= Bymbel + Gincee the last Aymbel emitted by the ounce can be ARBAC. The past hintoy can be sepasertedl by 3 thater ene d8) each qv s Ayabele 4, te alphabet ~ A 2 Ya % 4p the system ie in shake one ayrntel emitted by Se source wos A, oes ee tel A pee bai eeg y and Debane te state one ik amile elt © with i ite Probabit-ty ee en steak HWnee © E em fe ability Ya - ler © and gees de tlhe ee it probabil Ya Dept of ECE,SJCIT 54INFORMATION THEORY & CODING (10EC55) Dr. S BHARGAVI, Professor % The glate brantikione and Aye bel arrestin can ales be Sihwhabed using < Bike dingnenn” « A tree diagram ia a planan graph phere be b abel intava) : can be rected Uf atthe me 4 the plowing fransifiens + a ee 231353 4 [ conbinueton 3 Page mot 2 52455 3s 1 3 lnweoRen Reba done > Le Pom) be the probabilety ao Beqperce mj the Aouree - slog Pim) 2 © le Gn = z P (roi) 199, . Oe ee 2 , No symbole » Thee Oy and Dept of ECE,SJCIT