CORRECTIONDATA

OBSERVER
SOURCE

?
MMM
TRANSMITTERRECEIVERCORRECTINGDEVICE

Fig.8Schematicdiagramofacorrectionsystem.

Roughlythen,Hy?x?receivingpointtocorrectthereceivedmessage.
istheamountofadditionalinformationthatmustbesuppliedpersecondatthe
Toprovethefirstpart,considerlongsequencesofreceivedmessageM?and
correspondingoriginalmessageM.TherewillbelogarithmicallyTHy?x?oftheMs

whichcouldreasonablyhaveproducedeachM?.ThuswehaveTHy?x?binarydigits
tosendeachTseconds.Thiscanbedonewith?frequencyoferrorsonachannelof
capacityHy?x?.
Thesecondpartcanbeprovedbynoting,first,thatforanydiscretechancevariables
x,y,zHy?x?z Hy?x??
Thelefthandsidecanbeexpandedtogive
Hy?z Hyz?x Hy?x?Hyz?x Hy?x Hy?z Hy?x H?z??
Ifweidentifyxastheoutputofthesource,yasthereceivedsignalandzasthe
signalsentoverthecorrectionchannel,thentherighthandsideistheequivocation
lesstherateoftransmissionoverthecorrectionchannel.Ifthecapacityofthis
channelislessthantheequivocationtherighthandsidewillbegreaterthanzero
andHyz?x 0.Butthisistheuncertaintyofwhatwassent,knowingboththereceived
signalandthecorrectionsignal.Ifthisisgreaterthanzerothefrequencyoferrors
cannotbearbitrarilysmall.

Example:
Supposetheerrorsoccuratrandominasequenceofbinarydigits:probabilitypthat
adigitiswrongandq?1?pthatitisright.Theseerrorscanbecorrectediftheir
positionisknown.Thusthecorrectionchannelneedonlysendinformationasto
thesepositions.Thisamountstotransmittingfromasourcewhichproducesbinary
digitswithprobabilitypfor1(incorrect)andqfor0(correct).Thisrequiresa
channelofcapacity
??plogp?qlogq?whichistheequivocationoftheoriginalsystem.

TherateoftransmissionRcanbewrittenintwootherformsduetotheidentities
notedabove.Wehave
R
Hx Hy?x?
???H?x H?y H?x?y??
?

Hy Hx?y?
?

21
Thefirstdefiningexpressionhasalreadybeeninterpretedastheamountof
informationsentlesstheuncertaintyofwhatwassent.Thesecondmeasuresthe
amountreceivedlessthepartofthiswhichisduetonoise.Thethirdisthesumof
thetwoamountslessthejointentropyandthereforeinasenseisthenumberofbits
persecondcommontothetwo.Thusallthreeexpressionshaveacertainintuitive
significance.
ThecapacityCofanoisychannelshouldbethemaximumpossiblerateof
transmission,i.e.,theratewhenthesourceisproperlymatchedtothechannel.We
thereforedefinethechannelcapacityby
??

C
MaxH

??
?

x
Hy
x
?
?
?

wherethemaximumiswithrespecttoallpossibleinformationsourcesusedasinput
tothechannel.Ifthechannelisnoiseless,Hy?x 0.Thedefinitionisthenequivalent
tothatalreadygivenforanoiselesschannelsincethemaximumentropyforthe
channelisitscapacity.
13.THEFUNDAMENTALTHEOREMFORADISCRETECHANNELWITHNOISE
ItmayseemsurprisingthatweshoulddefineadefinitecapacityCforanoisy
channelsincewecanneversendcertaininformationinsuchacase.Itisclear,
however,thatbysendingtheinformationinaredundantformtheprobabilityof
errorscanbereduced.Forexample,byrepeatingthemessagemanytimesandbya
statisticalstudyofthedifferentreceivedversionsofthemessagetheprobabilityof
errorscouldbemadeverysmall.Onewouldexpect,however,thattomakethis
probabilityoferrorsapproachzero,theredundancyoftheencodingmustincrease
indefinitely,andtherateoftransmissionthereforeapproachzero.Thisisbyno
meanstrue.Ifitwere,therewouldnotbeaverywelldefinedcapacity,butonlya
capacityforagivenfrequencyoferrors,oragivenequivocation;thecapacitygoing
downastheerrorrequirementsaremademorestringent.ActuallythecapacityC
definedabovehasaverydefinitesignificance.Itispossibletosendinformationat
therateCthroughthechannelwithassmallafrequencyoferrorsorequivocation
asdesiredbyproperencoding.ThisstatementisnottrueforanyrategreaterthanC.
IfanattemptismadetotransmitatahigherratethanC,sayC?R1,thentherewill
necessarilybeanequivocationequaltoorgreaterthantheexcessR1.Naturetakes
paymentbyrequiringjustthatmuchuncertainty,sothatwearenotactuallygetting
anymorethanCthroughcorrectly.
ThesituationisindicatedinFig.9.Therateofinformationintothechannelis
plottedhorizontallyandtheequivocationvertically.Anypointabovetheheavyline

intheshadedregioncanbeattainedandthosebelowcannot.Thepointsontheline
cannotingeneralbeattained,buttherewillusuallybetwopointsonthelinethat
can.
TheseresultsarethemainjustificationforthedefinitionofCandwillnowbe
proved.
Theorem11:
LetadiscretechannelhavethecapacityCandadiscretesourcetheentropypersecondH.If
H?Cthereexistsacodingsystemsuchthattheoutputofthesourcecanbe
transmittedoverthechannelwithanarbitrarilysmallfrequencyoferrors(oran
arbitrarilysmallequivocation).IfH?Citispossibletoencodethesourcesothatthe
equivocationislessthanH?C where?isarbitrarilysmall.Thereisnomethodof
encodingwhichgivesanequivocationlessthanH?C.
Themethodofprovingthefirstpartofthistheoremisnotbyexhibitingacoding
methodhavingthedesiredproperties,butbyshowingthatsuchacodemustexistin
acertaingroupofcodes.Infactwewill

Hy?x?
ATTAINABLEREGION

CH?x?Fig.9Theequivocationpossibleforagiveninputentropytoachannel.

22
SLOPE=1.0

averagethefrequencyoferrorsoverthisgroupandshowthatthisaveragecanbe
madelessthan?.Iftheaverageofasetofnumbersislessthan?theremustexistat
leastoneinthesetwhichislessthan?.Thiswillestablishthedesiredresult.
ThecapacityCofanoisychannelhasbeendefinedas
C
MaxH

??
?

x
Hy
x
??
?
?
?

wherexistheinputandytheoutput.Themaximizationisoverallsourceswhich
mightbeusedasinputtothechannel.
LetS0beasourcewhichachievesthemaximumcapacityC.Ifthismaximumisnot
actuallyachievedbyanysourceletS0beasourcewhichapproximatestogivingthe
maximumrate.SupposeS0isusedasinputtothechannel.Weconsiderthepossible
transmittedandreceivedsequencesofalongdurationT.Thefollowingwillbetrue:
1.Thetransmittedsequencesfallintotwoclasses,ahighprobabilitygroupwith
about2TH?x?membersandtheremainingsequencesofsmalltotalprobability.
2.Similarlythereceivedsequenceshaveahighprobabilitysetofabout2TH?y?
membersandalowprobabilitysetofremainingsequences.
3.Eachhighprobabilityoutputcouldbeproducedbyabout2THy?x?inputs.The
probabilityofallothercaseshasasmalltotalprobability.
Allthe?sand?simpliedbythewordssmallandaboutinthesestatements
approachzeroasweallowTtoincreaseandS0toapproachthemaximizingsource.
ThesituationissummarizedinFig.10wheretheinputsequencesarepointsonthe
leftandoutputsequencespointsontheright.Thefanofcrosslinesrepresentsthe
rangeofpossiblecausesforatypicaloutput.
E

2H?x?THIGHPROBABILITY

MESSAGES

2H?y?THIGHPROBABILITY
RECEIVEDSIGNALS

2Hy?x?TREASONABLECAUSES
FOREACHE

2Hx?y?TREASONABLEEFFECTS
FOREACHM

Fig.10Schematicrepresentationoftherelationsbetweeninputsandoutputsina
channel.

NowsupposewehaveanothersourceproducinginformationatrateRwithR?C.In
theperiodTthissourcewillhave2TRhighprobabilitymessages.Wewishto
associatethesewithaselectionofthepossiblechannelinputsinsuchawayastoget
asmallfrequencyoferrors.Wewillsetupthisassociationinall
23
possibleways(using,however,onlythehighprobabilitygroupofinputsas
determinedbythesourceS0)andaveragethefrequencyoferrorsforthislargeclass
ofpossiblecodingsystems.Thisisthesameascalculatingthefrequencyoferrors
forarandomassociationofthemessagesandchannelinputsofdurationT.Suppose
aparticularoutputy1isobserved.Whatistheprobabilityofmorethanonemessage
inthesetofpossiblecausesofy1?Thereare2TRmessagesdistributedatrandomin
2TH?x?points.Theprobabilityofaparticularpointbeingamessageisthus

T?R?H?x?? Theprobabilitythatnoneofthepointsinthefanisamessage(apartfrom
?
2
theactualoriginatingmessage)is

THy?x?

T?R?H?x 2

P? 1?2
Consequently

NowR?H?x Hy?x?soR?H?xHy?xwith?positive.

approaches(asT?)
?
?

P? 1?2

?THy?x T??2THy?x?1?2?T??

Hencetheprobabilityofanerrorapproacheszeroandthefirstpartofthetheoremis
proved.Thesecondpartofthetheoremiseasilyshownbynotingthatwecould
merelysendCbitspersecondfromthesource,completelyneglectingtheremainder
oftheinformationgenerated.Atthereceivertheneglectedpartgivesan
equivocationH?x Candtheparttransmittedneedonlyadd?.Thislimitcanalso
beattainedinmanyotherways,aswillbeshownwhenweconsiderthecontinuous
case.Thelaststatementofthetheoremisasimpleconsequenceofourdefinitionof
C.SupposewecanencodeasourcewithH?x C?ainsuchawayastoobtainan
equivocationHy?x a with?positive.Then
R?H?x C?aand
Hx Hy?x Cwith?positive.ThiscontradictsthedefinitionofCasthemaximumof
H?x Hy?x?.
?

Actuallymorehasbeenprovedthanwasstatedinthetheorem.Iftheaverageofa
setofnumbersis
ppwithin?ofoftheirmaximum,afractionofatmost?canbemorethan?belowthe
maximum.Since?is

arbitrarilysmallwecansaythatalmostallthesystemsarearbitrarilyclosetothe
ideal.
14.DISCUSSION

ThedemonstrationofTheorem11,whilenotapureexistenceproof,hassomeofthe
deficienciesofsuchproofs.Anattempttoobtainagoodapproximationtoideal
codingbyfollowingthemethodoftheproofisgenerallyimpractical.Infact,apart
fromsomerathertrivialcasesandcertainlimitingsituations,noexplicitdescription
ofaseriesofapproximationtotheidealhasbeenfound.Probablythisisnoaccident
butisrelatedtothedifficultyofgivinganexplicitconstructionforagood
approximationtoarandomsequence.
Anapproximationtotheidealwouldhavethepropertythatifthesignalisalteredin
areasonablewaybythenoise,theoriginalcanstillberecovered.Inotherwordsthe
alterationwillnotingeneralbringitclosertoanotherreasonablesignalthanthe
original.Thisisaccomplishedatthecostofacertainamountofredundancyinthe
coding.Theredundancymustbeintroducedintheproperwaytocombatthe
particularnoisestructureinvolved.However,anyredundancyinthesourcewill
usuallyhelpifitisutilizedatthereceivingpoint.Inparticular,ifthesourcealready
hasacertainredundancyandnoattemptismadetoeliminateitinmatchingtothe
channel,thisredundancywillhelpcombatnoise.Forexample,inanoiseless
telegraphchannelonecouldsaveabout50%intimebyproperencodingofthe
messages.ThisisnotdoneandmostoftheredundancyofEnglishremainsinthe
channelsymbols.Thishastheadvantage,however,ofallowingconsiderablenoise
inthechannel.Asizablefractionoftheletterscanbereceivedincorrectlyandstill
reconstructedbythecontext.Infactthisisprobablynotabadapproximationtothe
idealinmanycases,sincethestatisticalstructureofEnglishisratherinvolvedand
thereasonableEnglishsequencesarenottoofar(inthesenserequiredforthe
theorem)fromarandomselection.
24
Asinthenoiselesscaseadelayisgenerallyrequiredtoapproachtheidealencoding.
Itnowhastheadditionalfunctionofallowingalargesampleofnoisetoaffectthe
signalbeforeanyjudgmentismadeatthereceivingpointastotheoriginalmessage.
Increasingthesamplesizealwayssharpensthepossiblestatisticalassertions.
ThecontentofTheorem11anditsproofcanbeformulatedinasomewhatdifferent
waywhichexhibitstheconnectionwiththenoiselesscasemoreclearly.Consider
thepossiblesignalsofdurationTandsupposeasubsetofthemisselectedtobe
used.Letthoseinthesubsetallbeusedwithequalprobability,andsupposethe
receiverisconstructedtoselect,astheoriginalsignal,themostprobablecausefrom
thesubset,whenaperturbedsignalisreceived.WedefineN?T?q?tobethe
maximumnumberofsignalswecanchooseforthesubsetsuchthattheprobability
ofanincorrectinterpretationislessthanorequaltoq.

?C,whereCisthechannelcapacity,providedthatqdoesnotequal0or

Theorem12:Lim
logN?T?q?T?T

1.
Inotherwords,nomatterhowwesetoutlimitsofreliability,wecandistinguish
reliablyintimeTenoughmessagestocorrespondtoaboutCTbits,whenTis
sufficientlylarge.Theorem12canbecomparedwiththedefinitionofthecapacity
ofanoiselesschannelgiveninSection1.
15.EXAMPLEOFADISCRETECHANNELANDITSCAPACITY
AsimpleexampleofadiscretechannelisindicatedinFig.11.Therearethree
possiblesymbols.Thefirstisneveraffectedbynoise.Thesecondandthirdeach
haveprobabilitypofcomingthroughundisturbed,andqofbeingchangedintothe
otherofthepair.Wehave(lettingplogp?qlogq?andPandQbethe

TRANSMITTEDSYMBOLS

p
qq
p
RECEIVEDSYMBOLS

Fig.11Exampleofadiscretechannel.probabilitiesofusingthefirstandsecond

symbols)
H?xPlogP?2QlogQHy?x 2Q??
WewishtochoosePandQinsuchawayastomaximizeH?x Hy?x?,subjecttothe
constraintP?2Q?1.Henceweconsider
UPlogP?2QlogQ?2QP?2Q?

UPUQ
??
??

logP2logQ
1
0
?
???

Eliminating?
2
2
??

2
??

0
?

logP?logQ
P?Qe?Q?
25

Thechannelcapacityisthen
?1P?Q2??2

??2C?log?
1

?Notehowthischeckstheobviousvaluesinthecasesp?1andp? .Inthefirst, 1
andC?log3,
2

whichiscorrectsincethechannelisthennoiselesswiththreepossiblesymbols.Ifp
1
? , 2and2
C?log2.Herethesecondandthirdsymbolscannotbedistinguishedatallandact
togetherlikeonesymbol.ThefirstsymbolisusedwithprobabilityP?1andthe
secondandthirdtogetherwithprobability
2
1

.Thismaybedistributedbetweentheminanydesiredwayandstillachievethe
maximumcapacity.2
Forintermediatevaluesofpthechannelcapacitywillliebetweenlog2andlog3.
Thedistinctionbetweenthesecondandthirdsymbolsconveyssomeinformationbut
notasmuchasinthenoiselesscase.Thefirstsymbolisusedsomewhatmore
frequentlythantheothertwobecauseofitsfreedomfromnoise.
16.THECHANNELCAPACITYINCERTAINSPECIALCASES
Ifthenoiseaffectssuccessivechannelsymbolsindependentlyitcanbedescribedby
asetoftransitionprobabilitiespij.Thisistheprobability,ifsymboliissent,thatj
willbereceived.Themaximumchannelrateisthengivenbythemaximumof
?PipijlogPipij?Pipijlogpiji?jii?j

wherewevarythePisubjectto P
i?1.ThisleadsbythemethodofLagrangetothe
equations,

psjlogj
piPipij

Pipit

Chststsjsj
hplogpststsjsj
hexpPiit
?

Ch
?
?

exp
sj

??

s
?

1
?

2
?????

MultiplyingbyPsandsummingonsshowsthat C.Lettheinverseofpsj(ifit
exists)behstsothat
shstpsjtj.Then:Hence:
or,

hstpsjlogpsj?logPipit?Chst?s?jis
hi

hplogphi
?iss?j
?tss?j

ThisisthesystemofequationsfordeterminingthemaximizingvaluesofPi,withC
tobedeterminedsothat P
i?1.WhenthisisdoneCwillbethechannelcapacity,
andthePitheproperprobabilitiesforthechannelsymbolstoachievethiscapacity.
Ifeachinputsymbolhasthesamesetofprobabilitiesonthelinesemergingfromit,
andthesameistrueofeachoutputsymbol,thecapacitycanbeeasilycalculated.
ExamplesareshowninFig.12.InsuchacaseHx?y?isindependentofthe
distributionofprobabilitiesontheinputsymbols,andisgivenby? p ilogpiwhere
thepiarethevaluesofthetransitionprobabilitiesfromanyinputsymbol.The
channelcapacityis
??
??

Hxii
MaxHThemaximumofH?y?isclearlylogmwheremisthenumberofoutput
symbols,sinceitispossibletomake
themallequallyprobablebymakingtheinputsymbolsequallyprobable.The
channelcapacityisthereforeC?logm? pilogpi?
?
?
??
?
??
?

y
MaxH
y
plogp
26
?
???

???

???
???

???

???

??????
???

???


???

abcFig.12Examplesofdiscretechannelswiththesametransitionprobabilitiesforeach
inputandforeachoutput.

InFig.12aitwouldbe
C?log4?log2?log2?Thiscouldbeachievedbyusingonlythe1stand3dsymbols.In
Fig.12b

InFig.12cwehave
C?log4?2log3?1log633
?log4?log3?1log23
5

?log123?3

C?log3?1log2?1log3?1log6236
3
?log?111

223366

Supposethesymbolsfallintoseveralgroupssuchthatthenoisenevercausesa
symbolinonegrouptobemistakenforasymbolinanothergroup.Letthecapacity
forthenthgroupbeCn(inbitspersecond)whenweuseonlythesymbolsinthis
group.Thenitiseasilyshownthat,forbestuseoftheentireset,thetotalprobability
Pnofallsymbolsinthenthgroupshouldbe
C?log2Cn?17.ANEXAMPLEOFEFFICIENTCODING
Cn

2Cn 2

Pn?Withinagrouptheprobabilityisdistributedjustasitwouldbeifthesewerethe
onlysymbolsbeingused.

Thechannelcapacityis
Thefollowingexample,althoughsomewhatunrealistic,isacaseinwhichexact
matchingtoanoisychannelispossible.Therearetwochannelsymbols,0and1,
andthenoiseaffectstheminblocksofsevensymbols.Ablockofseveniseither
transmittedwithouterror,orexactlyonesymbolofthesevenisincorrect.These
eightpossibilitiesareequallylikely.Wehave
?

??

181

?MaxH?yHx?y?

7788
??4bits/symbol?

log

Anefficientcode,allowingcompletecorrectionoferrorsandtransmittingattherate
C,isthefollowing(foundbyamethodduetoR.Hamming):
27
LetablockofsevensymbolsbeX1?X2X7.OftheseX3,X5,X6andX7aremessage
symbolsandchosenarbitrarilybythesource.Theotherthreeareredundantand
calculatedasfollows:
X4ischosentomake X4?X5?X6?X7evenX2 X2?X3?X6?X7X1 X1?X3?
X5?X7
Whenablockofsevenisreceived and?arecalculatedandifevencalledzero,if
oddcalledone.ThebinarynumberthengivesthesubscriptoftheXithatis
incorrect(if0therewasnoerror).
APPENDIX1THEGROWTHOFTHENUMBEROFBLOCKSOFSYMBOLSWITHA
FINITESTATECONDITION

LetNi?L?bethenumberofblocksofsymbolsoflengthLendinginstatei.Thenwe
have
NLNji
whereb1ij?b2ijbmijarethelengthofthesymbolswhichmaybechoseninstatei
andleadtostatej.ThesearelineardifferenceequationsandthebehaviorasL?
m
ustbeofthetype
Substitutinginthedifferenceequation
or
Forthistobepossiblethedeterminant
logAWLj
D
ThequantityCisthengivenbyC
??j
bij

ijij

WmustvanishandthisdeterminesW,whichis,ofcourse,thelargestrealrootofD?
0.
?

as
W
?

L
?i?s

b
?

?
?
bis

??
??

Nj?AjWL?
L?b

?s?AjWL?AiW
?s?Aj?AiW

iji?s

?bij

s?i?
s

logWandwealsonotethatthesamegrowthpropertiesresultifwerequirethatall
blocksstartinthesame(arbi
Lim
?L?L

?
??

Aijiji
?

0
?
j

???
???
sij

?
?

trarilychosen)state.
??

APPENDIX2DERIVATIONOFH p ilogpi
111
m.Fromcondition(3)wecandecomposeachoicefroms

LetHnnn
n
Abilitiesintoaseriesofmchoicesfromsequallylikelypossibilitiesandobtain
equallylikelypossi

?
?

A?sm mA?s??28
Similarly
A?tn nA?t Wecanchoosenarbitrarilylargeandfindanmtosatisfy

m n ?m?1? Thus,takinglogarithmsanddividingbynlogs,
?
s ?t ?s

?
?
? m logt ? or
? ?

nlogswhere?isarbitrarilysmall.NowfromthemonotonicpropertyofA?n?,
mlogtm1???nlogsnn

Hence,dividingbynA?s?,
?

m1 ? m A?t? ?
? ?

???

A?sm A?tn A?sm?1?mA?s nA?tm?1?A?s??

mA?t
or

nA
s
s
Klogt
?
?

nnnAlogt
?
?

A ? ? ??
A
ts
2logs
?

A
t
??

whereKmustbepositivetosatisfy(2).Nowsupposewehaveachoicefromn
possibilitieswithcommeasurableprobabilitiespi?
theniareintegers.Wecanbreakdownachoicefromnipossibilitiesintoachoice
fromnpossibilitieswithprobabilitiesp1pnandthen,iftheithwaschosen,a
choicefromniwithequalprobabilities.Usingcondition(3)again,weequatethe
totalchoicefrom n iascomputedbytwomethods
?

nini
where
Klogni?H?p1pn Kpilogni?
hi

Hence
Ifthepiareincommeasurable,theymaybeapproximatedbyrationalsandthesame
expressionmustholdbyourcontinuityassumption.Thustheexpressionholdsin
general.ThechoiceofcoefficientKisamatterofconvenienceandamountstothe
choiceofaunitofmeasure.
APPENDIX3

THEOREMSONERGODICSOURCES
IfitispossibletogofromanystatewithP?0toanyotheralongapathof
probabilityp?0,thesystemisergodicandthestronglawoflargenumberscanbe
applied.Thusthenumberoftimesagivenpathpijinthenetworkistraversedina
longsequenceoflengthNisaboutproportionaltotheprobabilityofbeingat
i,sayPi,andthenchoosingthispath,PipijN.IfN
islargeenoughtheprobabilityofpercentageerror??inthisislessthan?sothatforallbut
asetofsmallprobabilitytheactualnumbersliewithinthelimits
?PipijN?Hencenearlyallsequenceshaveaprobabilitypgivenby

Kplogniiii
i

H i ii
?

plogn
Kplog
nni
??

ij29

?
Pp?iij

Kplogp
?
Np

logpN
or
and
islimitedby
pij.Therateinquestionis
?

FN
andGN
monotonicdecreasing.Alsothey
logp

P ipijlogpij

N ? logp
?

?
?
?

??

N
Pipijlogpij
ThisprovesTheorem3.Theorem4followsimmediatelyfromthisoncalculating
upperandlowerboundsforn?q?basedonthe
possiblerangeofvaluesofpinTheorem3.Inthemixed(notergodic)caseif
L?piLiandtheentropiesofthecomponentsareH1?H2Hnwehavethe
logn?q?N

Theorem:LimN?
q?isadecreasingstepfunction,

s?1s??q

Hsintheinterval?i?q??i?11

ToproveTheorems5and6firstnotethatFNismonotonicdecreasingbecause
increasingNaddsasubscripttoaconditionalentropy.AsimplesubstitutionforpBi?Sj?
inthedefinitionofFNshowsthat
FN?NGNN?1?GN?1
N
N?

MAXIMIZINGTHERATEFORASYSTEMOFCONSTRAINTS
Supposewehaveasetofconstraintsonsequencesofsymbolsthatisofthefinite
statetypeandcanbe
?s?representedthereforebyalineargraph.Let?ijbethelengthsofthevarioussymbols

thatcanoccurin

?s?passingfromstateitostatej.WhatdistributionofprobabilitiesPiforthedifferent

statesandpijfor

choosingsymbolsinstateiandgoingtostatejmaximizestherateofgenerating
informationundertheseconstraints?Theconstraintsdefineadiscretechanneland
themaximumratemustbelessthanorequaltothecapacityCofthischannel,since
ifallblocksoflargelengthwereequallylikely,thisratewouldresult,
andifpossiblethiswouldbebest.Wewillshowthatthisratecanbeachievedby
properchoiceofthePiand?s?
andsummingthisforallNgivesGNmustapproachthesamelimit.ByusingTheorem3
weseethatLimGN?H.
1 F.HenceG
?

nN
APPENDIX4

? P
p

?s?

?s?

logp

iijijN

i ?s? ?s?ijij

P p ?
??

M
?s?

?s?

?s?

Let?ij?s? .Evidentlyforamaximump ?kexp? .Theconstraintson

maximizationare P
i?ijijij
?Pipijlogpij
Pi??ipij??jPi?pijij?

1,jpij?1,Pi?pijij 0.HencewemaximizeU?
Pipij
?
ij
i

UMPi?1?logpij NPi?ijiiPi?0?

pijM2
30