INTRODUCTION TO RANDOM SIGNAL ANALYSIS AND KALMAN FILTERINGA ee He anny ncn eso an ny ‘Scfemtnon apne oe Wy & So ar of Cones Cason eae a 1 Sgathey (Tommi) 2. Rado Preface Almost ll engncers zt invotved wit instrumentation and signal processing at thine ine nthe career. Along wih sian processing comes nine—thuy, important to understand it and know what to do about. Notlie rg re Formally dsseibed as random processes, and a considerable amount f teary ‘hout these processes hasbeen developed ove the past 40 years Te pinay fmphass here Ison linear least-squares filtering. Tis sect has become {specialy important since the recursive form of he filer was ntosused by ARTE, Kalam 1960, Therefore, much ofthe bok s devoted to ate ha ow known as Kalman tering. One of my purposes in wring this book was 0 tring doen the level ofthis subject and make available fo 8 wider audience, Thus. wherever possible, inutive ariments are used insted of ripoeons Jrock, and long derivations are svotiedwnless they ae erential to fll Sadestnding of the Litatios of the theory. Tete stone emphasis on examples. They reinforce the theory an demonstrate is applicable Ite engincerng problems “Te sti a andom processes and lina fering sno more cl than many other sbjects tat ure taught at the senor level in engceing. Yet hierarchy of prerequisite materials required and, fr this reason, the subject ‘sully ges pushed into the Deeianing graduate level. In pric a working nowedge of linear system theory is nese. This inches both Laplace aad Fourier tasform methods anc least, some acquinance with sta space ‘methods. The kindof restmen given na typical senor level cures ner Coated ystems guile adequate forthe level of mate presente here. Noise mt be described in probabilistic tems, so that a least a elementary Asowiedge of peobabity is Fequred before proceediag on To random process ‘hoor Since probabil ie «sibjec tht engincere often miss indeed Mes, Chapter | fils th gap. Ie "no fle” treatment that provides the ‘Socials needed forthe remaining sheper. Random signals and nc tng intrdscipinay. an this book can te sey all engineers and applied scientists not jst lectcal engines. 1 hve aught the material to mixed groups, and I fund that stages who ae not ‘lesa engineers fre just a well letra engineers. ‘The amount of ‘neil hat ean be covered in one semester, of course. depends om the back ‘Bound ofthe cas. Irs fst necessary to bang the group "upc seed” on obabiiy then sfc to squeeze al of the material xo One Semester.consider Chpiers § and 9 to be optional waters; they can be mite if ‘ecesrary. Aso, some insractors may peer to skip Chapter 4 on Wiener Iterne an go diet to Kalman ering. The book i onan so tis ay te done ithe ows n comma. Int dig ape, Kalman fring {obably the more important ofthe tw subjects: consequently if something Eicto ave, shoud be Wiener firing. A minimum pedspoical ote Would fe wcover Chapter oh nd Section Ie and do wel conser Mactobe the mow important matrat inthe book fom a aplication ew ‘As you ca see the book maybe used text in many ways, depending ‘onthe backpround and iret ofthe css. hope the level and ile wi a8 eat to enginersinindsiry ay acta seference- Kalman rng an ‘Specialy important topic with many potential appiaions, andi hin seo ony Sev! engineer ih the wu Bacrouné nates sd tae Sytem aay Tam rtf to my colleagues and stadent or heir encouragement snd elf sugeston ring Ihe reparation ofthe manusenn ao thank the tte staff o he Departmen of Electrical Enginecin or thei Rep ine “MSsoom nts. ep thank Jeane Ge fo reparng th al mam Sie GROVER BROWN, Contents (CHAPTER 1, Probabity and Random Variables a 3 1 tn Random Sas ‘orate ota Sot and Conn Prtuay Nut Ranorm Varies Conlon, Covance an Othogmay Sim of tdepenert Random Varnes Teeny Toward (CHAPTER 2, Mathematical Despion of Random Signals a Protas Desc of Radon Fst Gavan Random Pres ower Spec! Dena Fusion Random Tegrph Wave Peorndom Seas 1st contents nas 2s CHAPTER 3. Response of Linear Systems 10 Ras Detrminion af Auocoretion a Spectral Deity Fnetins fo Experiment Dua Spin Teoe | Dace cer Pamrm ad Ps Fourier Testor Pare White Nose and Bude Systens Neier ni (CHAPTER 4. Wiener Ftorng gees ff Dptmiatn with Respect wo Parmeter ‘he Stony Option ProbleWetng Futon {he Nomssioary Problem Origa Matiep Wiener Fier Probl ‘Weshtng Function Approsc tothe Sampled Data Fier CHAPTER 5, The Discrete Kalman Filter 4 Simple Recusve Ear Euapie ‘The Cosson Proces and Condionl Density Viewpoint CHAPTER. Applications and Additonal Tpks on Discrete ah Firing ‘Seqieatal Precesine othe Measnememt Data 103 "0 i a 23 20 m 465. Power Sytem Rearing Aolcaion 8 Retin o Demin Lest Sqes 9 Telephone Land Forecsing Example 610 Detrmaistc lpas ER 7. The Contingous Kalman Fier 14, Ton fm Dr Cai Fe Eaton 73 Coneied Memiremet snd Process Noe Ta. Colored Memnement Noe 275 Satopia Eno Ana 4 Relay Beton Wiener nd Kala Fits CHAPTER 8. Discrete Smoothing nd Predict 1 Pedi CHAPTER 9. Additions! Topics on Kelman Firing 94 Lincareation of Nowtiear rocess Dyas 32 Adspive Slew Kalan Fer 83 Squire Root Fen 34, Converion of Contnaous Measurements to Dist Frm—The Dele State Fier Ate 45, Stchaste Linear Replat Prem ab he Separation Toren APPENDIX Laplace and Fourier Transforms AA The Oi Late Tom [AS Toad apace Tanto INDEX 2 n Hs Ey Eo aa yo mINTRODUCTION TO RANDOM SIGNAL ANALYSIS AND KALMAN FILTERINGCHAPTER 1 Probability and Random Variables LI RANDOM SIGNALS. ‘Nearly everyone has some notion of random o oiselite signals. One hs oly tone an onary AM radio aay fom tation, crm up the Ylume and the ‘esis stati, oroise Ifone were t look st srip-char recoding of sach Sigal, it would appear to Warder on aimlessly with mo apparent orden is {ampli pattern as shown n Fig 1-1, Signalsf this type cannot be dseribed ‘rah expt mathematical fuscttons such ns sine waves, ep funtion, and the tke Their description ust be pu in probit terms. Early investi tors ecogized that andom signals could be described loosely i rms of their ‘pect content, ita rigors mathematical description of sich sta was ‘ot formatted un he 194, moat notably withthe work of Wiener and Rice 2. ‘Noises usually unwanted. The ative noise inthe aso signal dsurs ‘urenoyment ofthe mui or interferes with ovr understanding af the spoken Stor ase nan eecronc navigation sytem induces positon eros that can te dssserus in rfialsuatons; noise na dial daa transasion sytem an cause bit erors with obvious undesimble consequences, and onan 0, ‘Any noise that corps the desired sig is adi just a question of bow tad! Even afer the designer has done his best 10 eliminate ll the obvious oie producing mechani, thee always seems tobe some nose Ll! ver that must be suppresed with more sube means, such a Aitering. To 4050 tffetvely, one must understand nose in quantitative terns, ‘Probaility plays a key role inthe description of noslike signals. Owe Spits needs of subsequent chaps. The scope this insted inthis ear We make no apology for ths, because many fne books have been wien oa ‘probability in the breader sese. Our min objective hee Is the stu of rat ‘dom signals and optimal firing, née wh fo move on to ths area ‘quickly spossibe is, hough, ve must al least revi the bare essen c ‘Probability theoryFig 1. Typical mise actu 1.2 INTUMTIVE NOTION OF PROBABILITY Most engineering and scence students have had some acquaintance withthe fate concept of probaby Typically. withthe tute approach We fist ‘onside al possible outcomes of «chance experiment as Being cally ikl, Sod then the probability of parila evest say event Ais defined 2s “Possible outcomes favoring event A Toil possible uteomes where we read PUA) as “peobabilty of event A.""This concent is then expanded Toinside the reative requeney-oF occurrence or sata viewpoint of prob- sity. With the rove frequency concept. we imagine a large nuber of trals of some chance expermest and then define probability asthe relative Frequency of occurrence ofthe event in question, Consieratons such as what ismean by “lage and the existence of mis are normally avoid in lemen tary tealments This for good reason. The tea of imi in proba seme be ‘Akough the oder ative ncions of proabiity have imitations. they sti playa dnportant oe in probably theory. The rant possiblecvens oneep sa useful problem solving fol in many instances. The relativese- ‘quency concepts eapcily elf i visualizing the stastcalsigicance of the reals of protubity calcuation, That sf provides the necessary He tewcenthe theory an hepsi stustion. Two examples that usta he efucn of thes inttve nations of pabablty shoul aow prove useful may aan Example 1.1 In trig poker, each players dal S cals face down om 1 otk of 2 playing cards, We pose to questions: (a) What ithe pratt af being deal four fa kind, hat is, four aes, Four kings, an sofort? (oy Whur fhe probity Being dealt seit ash, tat i, acon fs sequence of ecards many it? Smunos 10 Qvesnosta) This problem is eave complicated if you think in tem ofthe sequence of chance events that can take place when the cars tedeatone at atime. Yel the problem atively easy when viewed rms ofthe aio of favorable to total number of stcomes. These ae eaily counted {in this ease. There are only 48 posible hands containing 4 aces nother 48 containing 4 kings; ec. Thus, there are 1348 pore four-of<-kind hands ‘he total numberof possible poker hands of any Kind i obtained from the combination formula for" 3 things aken ata time”" (3). The ten bythe ‘nomial coefclent, 2). St) st 50-9 (Sle mgrisy sts 2m 022) ‘Therefore, the probability of being deal four of «kind is Four of a kind) = Fz, = St om” Zsa ~ 000 (123) SournortoQuesno(b)Agui, the direct itemization of favorable evens the Simplet approach. The posse sequences in each of four sui are ‘AKQUO, KQUIOS,-, SARZA. (Note weal the ace tobe counted either high orlow.) Thus, hore are 10 possible stright Rushes n each ut ncading the royal fh ofthe suit) ving a total of #0 posible sraight fshes, The probably of straight Moshi hen _Sragh Bashy xcos aaa ae Taw We notin pasing hatin poke a straight fash wins over four fakin and ‘ably so since ti the rer of thet hands. Example 12. Crpsisa poplar gambling ume payedia casinos though: out the werk (1). The payer rolls two dice and plays agaist the Bouse (oe the casino) the st ral 7 or, the player wins inmeditey ite 2 30F 12 the pave oes immediatly Ifthe stroll resin 4 8.68.9. 010, he layer continues troll uni ether the same nuber appears. wich comsittes {in, or a7 appeas, whic renin the player losing. Wat f the payers Probablity of winning when teasing the die? "This example was chosen to state the shortcoming ofthe ect count ‘he-oueomes appeach, Th this case, one cangot enumerate all the pore outcomes. For example, the pliers rst rol ie, he play’ coatings wnt ‘other outcome is reached. Presumhly, the rolling cold continuo int without a 4017 appearing. which what x equred to terminate the fame Ths, the direct enumeration approach fs in this station. On the other hand, the relative frequency-ofeccuteace approach works qu el Table 1:1 shows the relative frequency of occurrence ofthe various mer on the first rll The numbers inthe column abled “pobabity were obtained by enamerating the 36 posible otcomes and alting for each utcome hat“ble 1 Protas Craps fae = = 7 pi — os nt = a 3s toe a ree =F te et oe cet ae ee Sh e * rer netore 6) = A ose . rw ‘ me cae rot = Am ee ttre = aoe eee) = wicaca comin a Fo ptay owing =H a8 sick sm conresponing tothe number athe fst column. For example. a4 tray be obtained with the combinations (1.3). (22), 0€ (13). For the cases there ihe gume continues sfler the st thw. the subsequent probabilities rere obtained singly by observing the lave eguency of wecurtence of the Thumbers volved, For example, 27s ice a Hkly aa 4. Ths, the rlative frequency af slling a? lore sho be twice that of before 7." andthe respective prbabiies are and 1. The total probability of winning with 2400 thet throw was resvoned a follows, A nly appears onthe est rll oF the time; and, of his rection, nly of he time wil this result in an ultimate win. Thus, the relative fequeney of winning vi this route isthe product of 4.71. Adit, this ine of reasoning is quite intuitive, bot that the very ‘ature of the relavedtequencyofcccurence approach to probably For the benefit of those wh ike to gamble, should be noted that craps is 1 very close game. The edge in favor of the house is only about It percent (also see Proem 13) i 13. Axiomatic PROBABILITY shouldbe apparent thatthe initive concepts of probably have thee init tins, The rato outcomes approach requires the equa ikelhood assump. tion fr all outcomes. This may fit many stations. bat often we wish fo Consider “unfai™ chanee situations aswell a "Ta" goes. Also. a8 demos stated in Example 12, there are many problems for which all posible cut ‘comes simply cannot be enumerated, The reaive-requency approach is int tive bya very nar. Inttion shoul never be ignored tt, onthe other and, it can lead one astray in complex stations. For these reason, the sxiomaie formulation of probably theory is now sou universally favored ‘among both appli and theoretical scholars inthis area As we woul expel ‘siomatie probably Ts compatible with the ode, more heuristic probably “Axiomatic probability hepins wit the concep of a sample space. We fst imagine «conceptual chance experiment. The sample space i the Selo ‘osnbe outcomes ofthis experiment. The individil outcomes are called ee- ‘ments x points nthe ample space. We dente the sample space as Sand ts ‘tof elements a si) The numberof points nthe sample space ima be ite, coum iit, or simply ifn” depending onthe expe ‘ent under eansieration. A few examples of sample spaces shoul be help atihis point sample 1.3 The experimen: Make a singe dra fom a deck of 3 play ing cars Since there are 52 possible outcomes he Sample space coma 2 discrete points. Ife wished, we could enumerate them as Ace f Clubs, King ‘of Cats, Queen of Cas, nd no forth. Note tha the ports othe sare space Imtis case ate "things rot numbers. a Example 14 The expertment: Tw fide are thrown and the numberof ois on te op ofeach is observed. Tere ae 3 ascrete outcomes that ean be ‘enumerated a8 (11) (121-(L).- (6.3) (68) The st number in parentheses denies the numberof dts on de 1 and the second is he number on (de Thos 36 distinct 2 tuples desrbe the porsble oteomes, and our sample Space contains 3 points or elements, Note thatthe pols ia his sample space ‘tain the Mentity ofeach individual de andthe number of ts shown on is opti Example 1 The experiment: Two far diese thrown ag he sum of he ‘munber of dots is observed In this expert, we do nt wish fo retain the ‘leity ofthe numbers on each die only the sum sof interest. Therefor. i ‘would be perfectly proper to say the possible outcomes ofthe experiment are (2.3.4.3, 6, 7.8.9. 10,11, 12. Thus, the sample space would contain 11 ‘dicrete elements. From this andthe preceding example ican be sen thal we fave some discretion in how we define the sample space corresponding {0.4 Certain exporinent, Ht depends to some extent on What we wish to observe. I Certain deta ofthe expeinent are not of trent they often may’ Be Sup- presaed with some reslantsimplifeation. However, once we ape on what ems are t be erowpe together and called outcomes the sample space most ncn all he dtned outcomes: end sma, te result o an experiment tun always yield one of the defied cutcomes, ‘Example 146 The experiment: Ada is thrown at age and the location ofthe hi obverved, In ths experiment we imagine the random mechanisms Lectng the throw ae sch that we pet coatnuous spread of data centered [round the bul'-eye when the experiment is repeated over dove. this tise, even we bound the ht locations within a certain region determined by fensanatleness, we sil caanot enumerate al possible hi locations. Thus, we fave an nite mamber of pins in our same spac i this example. Even ‘hough we cannot enumerate the pits one by ome they aro course, et ahiein rms of ether rcangulro¢ polar coordinates. Ml sume thatthe ml et isa subset of every se, and that very set i aubuetof ise. ‘One other comment about events isin order before proceeding othe bask anioms of probably. The event Ain sid to occur f any pnt nA coca. The thre axioms of probability may now be stated. Let § be the sample space and Abe any vent Gefined onthe sample space. The frst two stim are Asiom Ii PUA) = 0 3.0) Axiom 2: PIS) = 1 «39 Now. let Asa. Ay... Be mutually excarive disjoint) evens defined on S “The sequence maybe rite or countably init. The tid axiom then Adiom 3: PAU AVAL») = FAY PUA) + PUA) + ‘Axiom 1 simply says ha the probability ofa event cannot be negative ‘This certainly conforms to the relative fequeney-oFaccurence concept of okay. Axiom 2 says thatthe event, which incines all possible out omes, must have 4 probability of unity. Kt sometine called the certain vent The fis iwo axioms are obviously necessary if alomati probabity tobe compatible with he eer relative freqoeny potty theory. The hid ‘axiom f not quite so obvious, perhaps, and tsmply must be eSsumed. Ia ‘wor it says that when we have noaoverlapping (ji event, the proba bit of the union ofthese events the sum othe pababities ofthe inde val events. IFths were nwo, one co eel thik of counterexamples tat ‘would not te compatible with the relative frequency concept This weld be We now recapitulate. There are the essential ingredients in the form approach to probably Fist a sample space mst be defined tha ncades ll possible outcomes af eur conceptual experiment. We have some isrtion in ‘hat we cll outcomes, but caution i 8 order hee. The outcomes must be Aisin and allinciuive such that (8) ~ I. Second, we mst carefully dene 8 Sel of events on the sample space and the set rst be closed sich thatthe “peratins of union intersection, and complement also sil events othe set. Finaly, we must assign probaiies to all events in ascordance withthe basic ‘sioms of probably. In physical problems. his asgnment i choxen 1 be Compatible with what we fel to be emonabl in terms of elativefeauensy of toecurence of the events. Ifthe sample space § contains «fine numberof ‘ements, te probability assignment is uually made directly on the lence ofS. They are, of course, elementary events themselves. Ths. slong ith ‘Axiom 3. then inet assigns a probability tal ater evens defined onthe Sample space. However ithe sample space consist fan nite "smear of Points, te probability assignment must fe made on event ar non pos it the sample spoce. This wil be illustrated ater in Example 13, 39)‘Once we have specified the sample spe, the st of vents, and he proba tities associated withthe events re have what known as probly nace This provides the theoretical stature for the formal solution of a wide ‘arity of probability problems [Example 1.7 Consider a sing throw of two dice, and let us say we are ony interested in the sum of the ds that spear onthe tp faces. This chance Saaton ts many games tht ae played with dice In this ase, we wil de Sermaple oo tobe $= (23,45.6.7,8,9, 10,11, 2) and its seen to contain 1 dncret pints. Next, we define these of possible ‘ven tobe all sets ofS, nclading the mal et and $ isl, Note tha the ‘ements of Sar elementary evens, and they are disjoint a they shouldbe. ‘Aco, 1S)» 1. Fall, we nced to assign probabties to the evens. This oul be done atitrany within the contains imposed by the axioms of ‘robbly), but in this case we Want the resus of ou femal analysis to once wth the relaveequeney approoch. Therefore, ve wll assign pob- ‘hls to the ements accordance with Table 1-2, Which in tre inecty Spee pobabities fo al other evens dened on S. We nw ave a propery ‘Eined protblity space, and we ean pose avait of questions eative tthe Single throw of toe ‘Soppone we ask: What ste probability of throwing either 7 or am 11? From Aviom 3 and ong hat "7 or I" is the equvalent of saying "7 U rown=$o3 3a “Title 12 Protaiies or gue 12. Ven dings foro eens Aad Next, suppose we sk: Whats the probity of mt throwing 2,3 or 122 ‘This cals Tor the complement of event "2 or 3 of 12” which ie the Set {45.67.8101}, Recall hat we say the even oecarsifany clement inthe st ‘css. Therefore, again using Atom 3, we have Dees Se osseaese? ‘PINot throwing 2, 3.08 12) = ss) 5 Suppose we now pose the further question: What isthe probably that two" are thrown? In our definition of the sample space, we suppressed the ident ofthe individual ice, so ths simply Is not «proper aueston forthe ‘robablity space. as dened. This example will he continued, but fist we gress fora momen to consider ftrsccton of evens. Inaddiion tothe st operations of union and complementation, the oper tion of intersections also efi probability theory” The ntersecion of wo ‘vents and isthe vent containing poims tha are common tooth snd ‘This usrated in Fig -2 with wha is sometimes called a Venn dag ‘The points ing within the heavy contour comprise the union of A and denoted as A\U Bor""A oF” The pins within the shaded region re the ‘vent "A intersection 8 which is denoted A 8, or sometines ist And 1B" The following elationship shuld be apparcn jot fom the Eeometry of stonmatnot"Ab We nb roecangatey f se hthen Shem em oTthe Venn diagram AUB) = PA) + PIB) ~ ANB) 036 “The subactve xm nH (13.9 preset Because the probate in the esupping repo ave hen counted ice inthe wmmton of PA) and = "he probability PLA“ B) is known as the joint probability of A and B and witb uncused fhe Sston We des for the moment hough Tn lok two exampe ample 11 conned) We etn the woes example. Suppose we dete erent aytnorings #5 6,0r7. Eve B lb tes towing 9, Yoo I. We mo pose to gestion: 15 wi the potabiy of cent "A sod we. A Bt (Wath ay fee Ae AUB Te cSmon bot even sand Sethe nomber 7 te oy common ements 1 Pan B= Pm =E “The anger to () cam be found iter by itemizing the elements in 4 U 8 OF from Ea. (13.6, Using Fa (03.8) leads to MAU B)~ PIA) + PL) RAO BD "6% “Tis is easily vite using te direct temization method. zante 8 Lets recomie the dthowingetgsrinet We mish cont tingle tan where were ciclo Bo eval Hite Sipe dart wn cc he yr oii edn SG te er iss We pone hele unio: Wha ie pos. 1 a the per wits ote Sle how Was ati te ample sce pts te wal (We sume tue paper et the wall) Th tee ea ie mb posto same ace 5 Matha bane tefl ie two evento he ame sac—variable space woall then be: ‘Random Variable Realization Probably eeoome 0 [Noe tha the probaitis are nt seated uniformly i the random variable Sc. Te ea rest ofthe mapping is a et of el numbers representing the pose realizations ofthe random varble anda corresponding et of probe Eres hac sum to unity Once this correspondence ha been etabshed the ‘rg smple space usually ignored. “The random variable of Example 115i an example ofa discrete random vail in that ts alowable realizations are discrete (ce, countable) rather than continuous ‘The asoclated discrete vet of probaiiis is sometnes re fevvalto asthe pohablty mass dsiution of Sexply probability dsribution "We also ave occasion to work with continuous random variables. AS a matter fac the usual slectroni oie that ie encounteredina wie varity of Arfications fof this type, ttf, the vllage (or curren) may assume a mingus ange of values. The corresponding sample space then also contains fninfinte umber of pats, so we cannot ssignprobiis directv onthe points ofthe sample space: this must be done onthe defied evens. We ill tins the spite poner example of Section 1 (Example 1.14). als trate how this done 'LeLX denote 2 continuous random variable corresponding to the ang position ofthe pointer after it stops, Presimably this could e any angle FEtwcen an! 2 sadans hersfore the rotabiy of exaety any porclar poston sor, Thus, we asign probaly to the event that the pointer ops witin certain angular rage, say between O and @ radians. al pos tons are equally hie. is reasonable Yo asi probable as follows a pee ia oe as pz eg Me of npn tl Funston sketched in Fig 1. The linear portion of the function Between 0 tnd de to the “egal key assumption. ‘The funtion skeched in Fig. 1.6 is known asa cumulative dstbution {anton (or ut prokaity distbutionTanction), and it smply describes the probability assignment ast reflects onto equvalen’ events ie the random vari be space. Specifically, the probabity distibation function assoctated wih the random vanble Xs dined a File) = P= 8) ua whee is a parameter representing a realization of X. ust as Inthe discrete ase, once this dsb f etablihed in the ‘andonevariale space. the orignal sample space is usally ioore. It shoulé te clear fom the definition that a probability dstbution fanetion aways has ‘he follwing propenies LF 0, a0 DEM L a0 5. FAO) is a nondecreasing fueton of @ ‘The information contined nthe distribution function (6. Fig 1.6) may ‘so be presented in derivative form. Specialy lta) be defined 4 HHO = Sh Fao) an The function fx) known a the probability density uncton associated with the andom variable X. The densiy funtion for the poi! example shown InFig 1.7. From properties 1.2, and jst ed forthe dsnibution futon, ‘Should Be apparent thatthe density function has the Tllowing properties 1. lis a nonnegative function 2 [saorde = 1 It should also be apparent from elementary caus that the shaded are shown in Fig 13 represents the probaly that Xs between and 1, an te separated by a iafnctesial amount, 8, the ate approximely Ji) Mand thas we have'the term proba densi.se ewe 17. Probahity density fore for pote exp “The probability density and distri functions are alternative ways of escribing the relative dsrbution ofthe random variable. Both functions are sof in tandom-vriablesmalj, ad you should lvays Keepin mind the ‘Serivuivefinegral relationship between te two. AS a matter of notation, we Sil normaly se as uppercase symbol for the dstibution function and the ‘Corresponding lwercate symbol forthe density faction. The subscript n each ‘ase indicates the random variable being considered. The argument ofthe Fantom fs» dummy variable and may be most aything. 18 EXPECTATION, AVERAGES, AND. CHARACTERISTIC FUNCTION, “The idea of averaging is so commonplace that i may nol seem worthy of clatortion. Yet, thereat sublets, especialy as averaging lates oprah bili. Thor we need to formalize the notion of average Perhaps the st hing fo ates that we always average over numbers and rot “things.” There no such thing s the average of apples snd oranges ‘When we compute a student's average grades, we do not avetage over A.B ‘ho: innend, We sverage over uerca equiateats ha ave been arbi ‘signed to cach grade: Also. the quotes being averaged may oF may Rot De overned by chance, In citer cies random or determin the average {he sum ofthe mimbere divided the number of auanites being averaged. In the random cat, he ample average or sample mewn of arandom Varah is ‘etd se eKits 1a 7 wan whee X,Y. - «are sample ealzationsobained rom repeated als of the ‘hance sustion under consideration, We wil re the terms average and mean Inrchangeshly, andthe acective ample serves a 8 reminder tht We are Svergig over fie suber of tse iE, B.D. Tn ihe study of random variables we sto Ike to consider the concept verge that would coor fran infie number otis Tht Kea east" the relavesequency concept of probability. This hypothetical average is called expected vue and is ally same simply relers to what one would “expect inthe typical statistical situation. Bepating with discrete probeb- fy, imagine a andom variable whose m posible realizations ae se. The coresponding probubites ar pp. Pf we have Nils, ere Wis re, we would expect approximately pyN'ns, pu 1% et, Ths ‘he sample sverige woul be m. +N, si asa) pe WN + ‘This suggests the olowing definition for expected value for he dserete probs: bit case Expected va of X= BUN = 3 ps, assy where ms the number af allowable vales ofthe random variable X ‘Similarly. fora comiuous random variable X,ne have Expected value of ~ £149 ~ f° afte) ds as should be mentioned that Eqs (1.8.3) and (1.8. ar dtetions an the arguments leading up to these dainitons were presented to ive # senile fatlomale forthe deiniions and nota proof, We cam ve these sme Bu ‘mens for defining the expectation os fuetan of X, a wel as for X. Ths we fave "Discrete case Beir = Specs) uss) Continuous case: 1900) = f° eye de 1186 ‘Asan example ofthe use of Eg (1.4), et the function a(X) he. uation {16 forte dincreteconntrpart By. (1.45] hen proves an expesson for the th moment of that bx Pes a sn ‘The second momento is of pel tee nit gen by ax = [etn ae assy ‘The fst moment is, of course jus the expectation of.X, which sls knownasthe mean or average vale of X. Note when the tr sample somite, we ‘tety assume that we are refering fhe hypothetical infite-sampl average. 'We also ave aceaion to lok athe tecond moment of "about the mean.” Tis quant called the variance of Xs is defined as Variance of X= BIG ECO) as9 Ina qualitative sense the variance of i a measre ofthe dispersion of X shouts mean. OF cour, the mean zero, the variance ential the second momen “The expression for variance piven by Eq (1.8.9) canbe reduced toa more ‘convenient compattonal form By expanding the qlanity within the Backes Sdthen noting ht the expectation ofthe sm he sum a the expectations. “This leads to Var X= 2x2 = 2K B00) + (BL9F = Bu) — (BF (.si0) ‘The square root ofthe variance io ofinterest, ait hasbeen given the ame standard deviation, hf Standard deviation of X= Voarince AX = (LEAD) ample 116 Let X be unforly dstribate in the terval (0 2), Ts lead othe probaly density function (see Example 1.1 1 < fr Se 0. tewhere Find th man vr, an snd devin of “he mean jn the expan of ¥ and 8 en by Ba, (1.84) 500) em t= = fae ro Now that we have computed the mean, we ar i 8 poston to find the variance fom Eg (1.810, vere [Poha-e = as ‘The standard deviation i nove just the square roo ofthe variance Standard deviation of X = VER asi9 . soni erect cin said wth te ran vate Xe uta = fst tunis Ie can be seen that (ist the Fuser transform af the probability density function wit reversal of sgn on w. Thus the theorems (ad tables) of Fourier ‘wansform theory can be used fo advantage in evahting characteristic Rn tion and their verses, “The choacterstc function i expecially useful in evaluating the moments ‘of XThisean be demonstated a follows. The moments of may be writen 35 0p = [af de 819 20) = [sj aan FEEL. = [.stanem a = [Sate a aay fe) = [LL vores ae] = [pence ae ase 1 ca be sen iat an! honao SL, san “Thus with the help of table of Fourie transforms, you can often evaluate the ‘moments without pearing he intgrations indice thei definitions. [See Problems 1-24() and 228 for applications f the characteristic fonction | 19 NORMAL OR GAUSSIAN RANDOM VARIABLES “The random variable Xs cll normal of Gaussian its probability density function 1 o1) Mache l-ghe~ ma] as Fe cory cas pa el Tews ei ee als och iy [eres 9a) Pee mini ae a9» Note tat the normal density function & completely specifed by assigning Immerial voles tothe mean and variance. Thus « shorthand notaon has ‘ome into common anage to designate a noaal radom variable. When We XN 0? 99) swe mean 3s normal wih its mea given by the fst argument in prentheses Sus warance by the second argubet. Aso, a a mater of terminology the {ers normal and Gaussian ae used interchangeably in deseribng orl ‘dom varies, and se wll make no distinction between The two. "The normal density and distbutonfnctionsare sketched ia Fig: 8 and Lb. Note that he desi fupcuon symmetric and peaks ats mean. Qual tively, then, the mean tescento be the mos likely valve, ith valves om ter tde of the mean gradally becoming les and les key the dance 19 NOMA ON oMUAAN HaNDOM VaMARLES 27 ey re 14 Noxmal densiy and dso aetns. Normal dems action. (8) ‘tom the mean becomes lager. Since many natural random phenomena sen 10 ahi this ceatatendeney proper) teas approximately, the arma ds {ebution is encountered frequeaty in applied probaly. Recall ha the va ance isa measure of dispersion abou! the mean. Thus sll oresponds oa Srp peaked deny curve, whereas large wil ye curve with 3 Mat peak ‘The normal dstabution function sof couse he iategeal of tbe density rey ff de =f on [humor] de 09.5) Unfortunately his iatgrl cantor be cepresented in closed form in terms of elementary fonctions, Ths, valve must he tabulated ssa function of he Upper lint A bre abun for ze mean and amity variance i give io “Tale 19" A quick pace st he ae wil show tat he dition onc very closet unity for valves ofthe argument greater than 40. our table, which was taken fom Feller (4. the taulation qu at abou this pit. and {hss adequate for mos purposes. Hosever, im some instances, the ference between Fy and unity, beitever so small, very much of nteres and we peed to be able fo compute it. An approximate formula for dong 50 for zero eansat ay vtec by Ic mye sume tenn fr, T cy Eq (1.96) only within x few percent for» equal 4 Te iproves Fpl ‘ough a xinereases, “Tie 1.9 The Normal Density an ise fine 7 fa 7 ‘ae 19 conte) 7 a re {is worth mentioning that tables of neal distribution ae presented in tapes in Fig. 1.12 au number separated by cornmas, We oul of couseMaou “ee. it and Margin Probie fr Xand YX Fest Drawand Yi Second Dra Kf sw | reer Inve shown them as two-lement coun vectors: ls purely 2 mater of nota tion fe assim that all elementary evens i the sample space ae equally kel, each shouldbe assigned x probably of The on probaly dstibu- ton for and Ys thee as shown in Table 1.10. Just sin Section 1.4, we can write certain relationships among the joint marginal, nd conditional probabies for random variables X ad 7, “Margit tuncondtonl probably: pd Zr) ana pri = E pata) (tay Condon proba oe ona) Boyes ae: ats) ‘The arguments in Ey. 114) 101. 11.6) were omitted because the permissibe sand are oboe The decree random varisles¥ and Yar defined tobe saa nde- endef rote 9) = pte) aL ‘caste any, vance hernia rane 9 tically independent beause al ine potas in Table 1.10 ait {he produc relationship even by Eq (1.117. Tit ws nv turn ourftenton to continuous random variables. Justa in ‘he siale-variable cose, the rlatve distin inthe lve case must Fe denrioed in terms o ether a cumulative dbo fnetion ora density Fanetion, Let and ¥ be continous random valables. The on cumulaive ‘herbi function is died” 2s Foe. 9)~ PWS and Y= 9) (uty teri, Fy baste folowing properties a Filon any Ft) = | at. © Firinconccenseginranty (LILI Tein sensi finn conn inim viesXand Ys Sr 9) = PE cans) ‘ote that there an ntopralidrivaive relationship between the cumulative Aixiibaion and densty fnctons, just a athe single-varate ease. Thus 10 Find she oat that x joint reaiation of X and Y wil he within a crisin ‘etion inthe ple, we use the Joe integral Formals Pex and Ye within 8) = [Jide yeeedy ALI) This is shown gometrcallyin Fig 113. Ofcourse, ifthe reson Ris diferen til rectangle, shown in ig 118, the proabityof8 and Y Ing whi he rectangle Is Pay 2X Sy + de and y= V3 9» fines sods dy ILI) “Thu it shouldbe apparent that fy has the wal meaning of density na two mensional sense. “The marginal or waconditonal densities ar obtained is 2 manne sila to the discrete case excem that summation replaced Wi iteration This6 Promanisr¥ Aso RANDOM VABARAS ge 1.13, Resin i le. we have it~ [ft ans 5:00 = ffs 9 cttt6) Eqotions (11.4) ad (111.8 frseret conditional rabies canbe app to dens function for ferential regions. This results in similar ‘mutoes for coniinal denies. The dfn regions showin Fg. 14 ‘ete used in the following development rom the basic defiion af conde Twa probably. ve have fink, sade eldest nim PX isin strip dl A stip Figs 146, Dire area ie 1 sot matoon vanes 37 ‘Canceling the dys and noting the statement “Yisinsrp dy isapproximately {he sme assaying" sequal fo" leads 1 yee Note that the quantity in brackets in Ea (111.18) has al she earmarks of « densy function. and ts probabilistic iatrpretation wn the lef ie of the ‘union, Thus we define condonldensy a, Pg 5 00+ del = 0 co) on) ‘The functional dependence onthe second argument y hs bee intetonally ‘mie in fy. This sto emphasize hat fy 3 deny fonction on foot Y ise Ba (11118), Ofcourse,» does appari fay a parameter, which may ‘though ofa given, deterministic quantity Ir ot the primary argument ffi though “The analogous dfition a yi foe.) tet foxes = BE 20 ‘Once Bas. (1.1.19) and (1.11.2 have Been established, Bayes rule flows Bayes ue fap, en Sue) = fae 2 Statical independence fr contnaous random variables X and Vis de fod inthe sane manne a for dscrete variates tat is X and Yar tas cally independent if foes 31> ed res Am example is now i onder Example 1.29 Consider a dart-hrowing ume in which the age con ‘enon ty coordinate system. The plyer aims cach thom atthe origin 36 Contig tos or er best abit. Since there are many vagaries affecting each ‘wowne can expects scatter nthe it pattern Alo afer some practice, the Seater should be unbiased, lftworiht und upand. down. Lette coordinate Oa hit bea bivariate randor variable cX- In this example we woul no. fxpect the coordinate fo affect # in any wy: therefore staisia indepen: ence of and Ys reasonable ansrmpion. Alo, because of the erial tendency in and ¥ he assumption of wormal jbusion would appear tbeare 1S, Sketch of one gut of ‘ ‘Nai nema ens ston reasonable, Thus we assume te ftlowing unconditional densities: ns hee tet ony Hoe ghee 20 ‘he ay ten eb Eqution (1.258 the speci case of bivariate normal densi function inwhih Xan Far independent, unbsed, and Have eaua variances. This fn important density funeton and ie skeched in Fg. 1.15. often de eae 116 tapine Een scribed asa smooth “hill shape function and, fo he special ease here, the hil is symmetric in every respet. Thos the contours shown in Fg. 11 are crcl cles “The joint densiy function fora discrete bivariate random viable X, 1) «an be represented with impulsive‘ype functions, just asin the sngle-ariate ‘Sine However in the bivariate ease, remember that volume (ot aes) rep sents probably. Thus volumeype pulse fusions are appropriate. When {Xand Fare independent, the appropriate twoimenionl imple function ‘htained as a simple product cf Dra deta fanclions. For example. uri “alume impulse atthe origin is sven by BBG). However, the station it ‘more complicated when X and Y are not independent nd we simply Pave 10 {maeine appropriate volume-tye impulsive freon. We will conse exanh ‘les of in subsequent chap The mpulive density fron represent tion forthe ot drawing example, Example 119, is shown fa Fig. 116 12 CORRELATION, COVARIANCE, AND ORTHOGONALITY ‘The expectation ofthe product of wo random va sntres In general. i ven by Mes and 8 of speci ur = [J fete ideo wa) ‘Theresa special simpicaton of Eg (121 that oscurs when and ¥ are independent In this case. fey ay be factored (ne Eq 11.20, Equation een = [ofteide[ sfonde = mie 4.122) a and posses the property of Ea (1.12.2, that is he exptation of the prods the product ofthe iil expectations, they aed to be Lincorrelted. Obviously X and Fare independent hear ls uncle tod. However. the converse aol tra. eseept ina few special cen ee ‘As mater of terminology if aun The covarace of and Fi hoof special interest and itis defied (Cov of X and Y= BIE = KP = mth unsWith he dfn of 9, (1.12.4) we can now define he cola coefent fortwo random variables as (Cow of and ¥ Corton cotiiet = p = SSA LT, B= mi — mi) ‘The correlation coefficient is normalized measure ofthe degree of correlation tetween to random vrahes, nd the nomaiation such that awa es srithinthe range =I'= = 1. This wil be demonstrated 0! proved) by foking thre special eases ans 1. Y= (warimum positive correlation: ‘When Y= X, Eg (1:12: becomes EX — mi = ma °° Tear = ma VER = mak 2. = =X (mavimum negative corelation) When YX Bq (L125) Becomes FI = my -X + mh VEX mat VERE + ma 3.AC and Yunconeated, thats, EUX) = BONEN) Expanding the numerator of Eq. (112.5) yields BUCY ~ mes ~ mo + mama) = BACT) ~ mg BUD) ~ my EC) + mxmy Thus p= 0. ‘We have now examined the extremes of posiive and neptive correlation and ‘zero correlation; there can be ll shades of era inbetween, [For frter de {als ee Papouts (1 1.13. SUM OF INDEPENDENT RANDOM VARIABLES, AND TENDENCY TOWARD NORMAL DISTRIBUTION Sine we frequently need to conser ative combinations of independent Fandom varsbls, thi ill mow be examined i some deta. Let X and Y be Independent random variables wath pobabity density functions J.) and ‘Fy. Define another random variable Zas the sum of X and: gexer ann Given the density functions of and ¥, we wish to find the corresponding density of Z Le: be a particular reaizatin othe random variable Z, an think of a8 bing aed. Now consider all pose reakalions of A and Y that il (lety. they satis the equation rtyee ana) salle is inte y ae ios sight shown nF Nest, cosider an igcrementl penurbation of ¢ 1 ¢ + ds and again ‘consider the locus of ealzatons of and tat wll yild z+ de. Tis oes is Sp stright line andi shown a he Gpper ie in Fig. 1.17 It shod be ‘pparent that ll sand ng within the ferential step beeen the two lines imap ito polts between: abd : + dein the: space. Therefore, Pes 252 + de) = Picand yin diferent sip) = Vie fenfdrrds dy Ba) But, within the dierent stp, ys constrained to according to ix a0) Also, since the stip s only of dierenial width, the double iegral of Eq, (0.133) reduces to single tera, Choosing ras the variable of ttgration nd ating that d= dead neszaced=[[nionte-ou]e an gee 1, ire stip ned fr de sietis now apparent from Eg, (1185) tha the quantity within he bracket i the desired probably density function for Z. Ts aes =f funte 9 ane sof tees to note that he integral om te vg side of Ea. (1.18. is convolution itera. Thus fom Founer transform theory. we ca then write FUG = U1 9A aan where} denotes “Fourier transfor of ("We now have wo ways of, austin the density of 2 (1) We can evaluate the convolution integral d fealy. of 2) we can fs ransform fy anf then frm the product ofthe ‘tansfarms, and fll invert the product to get fe. Examples that istate {ach ofthese metho ow. Example 121 Let Xand Ye independent random variables with identical ‘rectangular deasty functions as shown in Fig. lea. We wish 10nd the ‘dnsty function fr thew sum, which we wil ell ‘Note fs tha the density shown in Fig. 1.18 has even syrametry. Thus fds 0) ~ fae ~ 2, The convolution ig expression of Eq (112.8) 8 ‘hen the ltr of rectangular pulse multiplied by a simlar pulse shied to theright amount =. When 2> 2 or <2 thee is nooveriap in the pues s0 ‘hei product zero, When ~2 =2 = Ohere fsa nootrivial overlap Which ie ‘east linearly bepnning at: = ~2 and reaching maximum a =~ 0. The con ‘otto integral them increases accorialy at shawn Fig. 18h. A Sala Bal ee (era Fandom. ta Dey srgumeot may be used wo show that /(2) decreases inealy inthe itevat ‘uhoreO='2 = 2 This leads othe trangular density function of Fg 18. "We can go one sep frtber now and ook athe demsiycoresponding to te sum of thes random variables, Let We defined ae wextrey ane here X,Y, and V ae mutually independent and have ential rectangular Gems as shown a Fg sia, We have already worked ou the density for X "7.0 the density of Wis the convolution ofthe to futons shown Tig Litvand 1 IRD: We wl eave the detail of the as an exerci andthe rsa is shown i Fig. 1.19. Each ofthe segments lnbcled 12. and 3 fa ae of tarbola. Notice the smooth central endene. With lite mination oe en ‘cea simiarty betwen his apd 2 ea normal density carve If we were {ogoanother step and covelvea rectangular density With that of Fig. 119, w= ‘Wot el the density for the sum of four independent random variables. The ‘esting function would consist of comected Segments of cub functions ex ‘ending from "To #4 Ii aprearance. though pl shown. would ese The normal cuve even mote than ht of Fig. 1.19. Andon andon-each atonal Convolution resus ina curve that resembles the nonmal curve more closely than the preceding one “Trp examples intended to demonstrat not prove) that a superpo sisan of independent random arabes lv end toward normality. rears les ofthe diatnbution ofthe advil random vars contauting to the Sum. This f known asthe central fini theorem sais, Wi on remake therem snd ts vay subject o only modes restos Inengineering aplictions the aos ee must deal with reuenty di to ‘Sipeposition of many all costnbutions- When ths 40 We have good Temom to make the assumption of normality Te cea lini theorem says 0 oj that. Th we hve here oe of the ressons fr casein enter Std ners normal random varsaben™they are a common aecrence in wwe.Example 1.22 Let X and Yb independent normal random variables wih sero means and variances and. We wish nd the probity density fnetion forthe sum of ¥ and F which wil again be denoted as Z. For vary. ‘ella the Fourier transform approach. The explicit expresion fr fx tnd fy ate 10 = ii ans fin» inet ona Noe that we have sed as che dummy argument ofthe futons 1 of no onsequence because it integrated out inthe transformation to thew dom. ning Founer transfor aes, e Bathe transforms of fra tobe Fil = ev ana SU = 3a Forming thir prot ye HII = ein ana» “Ten the inverse pves the desired f: faa) = Fens, wa.) Note tht the dest function fr 2 x als nema in form, and its variance is ven by dnckt ot assy The summation of any suber of rom variables can sways be thought ‘of asa sequence of summing operations on two variables: therefore should be clear that summing any numer of independent normal random vases leads to anormal random saab. This rather emarkable result can be gene. alized utr to lala he case of dependent normal random rials, hich wee wil sous ter. LIM TRANSFORMATION OF RANDOM VARIABLES A mathematica transformation that takes one tet of variables sy, inputs) into Anaer st (ay. outputs) ts 4 common station fa sfstemsanaiysiy Let os begin with simple sinalesnpur single-output talon where the peop restioaship is governed by the algebraic equation y= a) cay Here we are fetrested in random inputs. 0 think of a elizationo he ‘put andom vale X, and yas the corresponding realization of the output [Atsume we know the probaly dens faction for X. ad would ike tf the eorrenponding density for. Iis emping osm replace x nF) with {soquialeat is terms ofy and pas oat tat, However ts not gi hat Sipe lhe seen presently. Tt et us scue that the transformation gs one-ane fr al pr snisible By this we mean tha the functioalreaionship given by (1:16) can be reversed, and canbe writen unigusy asa fenton of. Let the "revere relationship te The probabilities tha and Ye within corresponding frei regions must be equal That [UX is between and + de) = PAY is beeen and y +) 1.143) [°° fread da. tor dy positive [sean = an =[f hide, fod sepsive ‘One ofthe sults f this problem should noe paren from Ea (144, postive dr yes negative dy (ve nepative derivative. the negra of, runt be taken fom = to inorder oid postive probaly Tas the equivalent of inerchangng the nis and reversing ie ign shown Eq ties) “The differential equivslemt of Eq (1.144 is fae ds = fds vias: hore we have tact assomes di tobe postive. Alo. xis constrained 10 be or, eauivalent fs) = wfc re) where Ws) inicates the derivative ofA with respect ts, To examples il ow be presented