Professional Documents
Culture Documents
Problem Solvers Stochastic Processes - (Coleman)
Problem Solvers Stochastic Processes - (Coleman)
0, Pe Pift) = 1, ie. P(t) is a stochastic matrix. a (ii) A continuity condition. As t — 0, p;,{t) + 1; then by (i) p,{t) > 0 as 1 +0 for i # j; therefore p,{t) + 5, 4 ie. P(t) > Last +0. (iii) The Chapman-Kolmogorov equation holds, ie. P(u+v) = Pw)P(r) (u,v € [0, <0) If S is finite, then the unique solution of equation 6.1 is Pu) = e'@ = 3 war where Q is a constant matrix. Given Q we can find P(t), and given P(t) we can find Q. Thus Q can be used to ‘represent’ the process. é Ifa remainder term A(t) = O(t) as t + 0, then |A(0/t| < c, a positi constant, as t + 0. Then from equation 6.2, as 1 0 P(t) = 14+ 1Q+0(") Pift) = 5 +tq,,+O(0?) fg lt ee 4y = fig HEM =F oy0}| = HO) say since 5,, = p,{0) (by (i). If A(®) ~ ct as t+ 0, then A(t + ¢ as t> 0. Therefore, for i # j, p,jt) ~ qyt as t+ 0. Thus q,,t is the probability of a jump from i to j in short time interval t; q,, is termed the transition rate 6.3 we can show that if i # j then q,, > 0 and that q,, < 0. We th set q, = —qy 2 0. Problem 6.1 Let rv T, be the time spent in state i before a jump from it. Prove that 7, is &(q,). Solution. In Problem 5.6 we proved the special case for the Poisson process. For fixed , definear , tr 4 es er ae oe le _ PO) = pr{Z, =i at times t = t/n,2t/n,...,(n—I)t/n,t|Z, = {p,(t/n)}"_ by the Markov property and time independence = {1—g,t/n+O(P?/n)}" + ee asn + 0 asn + co, Pi) > pr{Z,=i (<< 2)} = pr; > 2). Therefore é pr(T; < t) = 1—e~*, the df of an 8(q,) rv. o _ If0 < q, < ~, then state i is called stable; if g, = 00, then i is called instantaneous; and if g, = 0, then i is absorbing. By Problem 6.1 a particle instantaneously jumps from an instantaneous state, and never leaves an The Kolmogorov differential equations relate P(t) to Q. The backward equations are 70 _ ape PA) = & Gu Prj(t) are generally valid. The forward equations are ah) _ simian vif0) = 5 Paley these do not always hold. 2 Applications _ Problem 6.2 The general time-independent birth and death process with mmigration. Consider the Markov chain {Z, :t € [0, o0)}, where Z, is the ion size at a time r. If Z, = i(i = 0, 1, 2,...), then in a short time nterval (t,t+A), Z, increases by one (by a birth or the arrival of an immigrant) with probability 4,4+0(4), decreases by one (by a death _ or the departure of an emigrant) with probability .,4 +.0(4), or does not change with probability 1—(2,+y,)4+0(4). Any other changes must efore have probability o(4). Clearly 4) = 0. What is the matrix Q? the Kolmogorov forward differential equations, and determine their Solution. The state space is {0,1,2,...}- Since the transition matrix P(d)is 2,4+0(4) (= i+) 1-G,+u)4+0(4) G =i 1A +0(4) G=i-1 0(4) (otherwise) (64) a 0 —W-A, A, ” 9 a a eal Let us abbreviate p,{t) to p,, and use pj, to denote (d/dt)p,(t). Then, i Z, = i, the forward equations are Pio = —AgPio + Hs Pix Pry = Aya Puy- 1-H Ay tHe rPajer UF = 12,.-) For the equilibrium solution, let p,{t) > 7, as t + co, then 0 = ~Agm y+ Hm, 0 = Ay 1-H) —Aj— Hyer) G=12.-) Write Ty Amy MyerF%er UV = 0,12.) Then =-l 0=1,,-1, G=1,2..) 80 1,=0 G=0,1,2,....) Therefore Rye /My = Altyes Therefore My = (y/my_ 1) (Hj /j-2)-+-( [ro)o = Vil where y= SoBe zeae Yom 1 Hgftj— yoo By Then, since Ex, = 1, l=1 pe so m= H/d% = 01,2...) This will be a probability distribution if and only if Ev, converges. Problem 6.3 The linear birth and death process with immigration. Con- sider a population process {Z,:t€[0,0o)}, where Z, is the size of the population at time t. People in the population behave independentlyibility -o(: eens Will give? Terie to “andthe” wasrnter: Pith bability 44 +-0(4) the member will die or emigrate, and with probability —G+n4+o(4) nothing will happen to the member. In 4 also, an m nt will join the population with probability «A +0(4). If initially - the population contains i members, what is the pgf, 17s), for Z,? Solution. We consider a decomposition based on the last 4. Then Zysa = Z,AX +. 4X, 4W we 1 ifthere is an immigrant in (t, t+ 4), i.e. with probability «4 +0(4) x '0 otherwise, ie. with probability 1—a4+0(4) 1 ifthe Ith member gives birth in (t,t + A), ie. with probability 4A+0(4) —1 if the Ith member dies or emigrates in (t,t+ A), i.e. with probability pA-+0(A) 0 otherwise, i.e. with probability 1—(2+u)4+0(4). Es” = ads'+(1—ad)s°+0(A) = 1—(a—as)4 +0(4) Es*| = 2As'+{1—(A+p)4}s°+pAs~!+0(A) = 1+ {4s—(At+p)+us™'}4+0(4) “eX (Es) (Es%). ..(Es*) (Es) = (Es™) Es”) = Been —(iA+ipta)A +0(A)} + {ind +0(A)}s~! +0(4) the expansion and collection of terms which are O(1) and O(4). lore Zi+4=5|Z, = )= p(X, +...+X,+W = j-i (iA+a)4+0(4) G =i+1) 1-G+int+a)s+o(4) ipA+o(4) G=i-1) (4) (otherwise) It is clear that we could have written this, or equivalently the Q-matrix, it down. It is of the form (6.4), with 2, = id+a, y, = in, and so quations 6.5 are (0 = — Pio + Pi Py = (i -A+a}p, 1. —UA+u) + a}p+G+ Dap. je. G= een Sh a = Oforj=. a hold for j = —1, 0, 1, 2,.... We 68 by s! Hs! = As? j— Isp, j_, +ass~'p, , ‘We then sum over j = Ts) = EzjzgniS* = >, p,(t)s! ai aes gpa an Gs Font +p Then ot és ie. oT asa eS aT ot és We shalll solve this partial differential equation for IT. We shall not dis here why the method works. We solve the auxiliary equations dt ds aq 1 @s—p0—9§ ~ —a(l-s We shall suppose 2 # 41 (Exercise 5 is the case 4 = 12), then a—yar = 24544 as. al ay ie a eee les As—u I-s (GES) - —p)"IT = constant Since the partial differential equation is of the first order, it can have o one arbitrary constant, so the two constants must be functions of another, ic, ——e7 G-#! = constant Therefore (s— wr = Afteoa We determine / from the initial condition: Z, .€. I 9(s) = (s—yys! = (22) By setting r = (4s—p)/(1—s), so s = (u+r)(Z+0), we determine th function f(r). This, when substituted into (6.10) gives I = I7,(s). osiren Z, = 4 Z; hs the ma dnibtion a 29-7 «-+¥?, where Z* is the number of immigrants who arrived in 0 d are still present at time t plus their descendants who are still present _ at time ¢, and 1 if the jth of the i original members is still present at time t iia 0 otherwise (i= 12...) Then Z} and each of the Y¥are mutually independent, and Z** is distri- like Z, given Z, = 0. Also, each Yis a Bernoulli rv with probability ~ of success, pr(T > t) = e™ (by Problem 5.5) Therefore Ts) = EAH Belew te! fe need therefore have considered only the case i = 0. special cases have been given names: A = yt = 0: the Poisson process _ 2 = 0: the immigration-emigration process a = = 0: the linear growth process or the Yule process a = 0: the linear birth and death process roblem 6.4 Find EZ, for the population process of Problem 6.3. Solution. We can find EZ, and VZ, from the pef IT,(s). However, it is ot necessary to solve equation 6.9 to determine them. If we differentiate equation 69 with repost to s, we obtain oF is— yo gett {As—w(-1)+A01-)— as—1} 2 oll (6.11) 1) =1, en tae - ari 5 1 in equation 6.11 we thus obtain a first-order ordinary dif- _ ferential equation in M,: Meu aM, =a (6.12) st mmmeaie Since en 2 (em d Hemel... 7 (3) | ae EZZ AD} = {vam M) =% z, See ae where V, = VZ,, we can differentiate a tidied-up equation 6.11 with respect. to s, and let s > 1, to obtain a first-order ordinary differential equatio for V,, into which we can substitute for M, from equation 6.13. Note the KW = VZ, = 0. Problem 6.5 The single-server queue with Poisson arrivals and exponen tial service times, Customers arrive in a Poisson process of rate 4, and join the queue. The service times for customers are independent &(ui) Determinea partial differential equation for I7,(3), the pef for Z,, the num! of customers in the queue (including the one being served, if any) at time Find the equilibrium distribution of the queue size. Solution. In (t,t +), pr(a new customer arrives) = 14 +0(4) pr(a customer finishes being served and he leaves) = A +0(4) pr(no customer arrives and none completes being served) =1- (i+masold) pr(anything else) = 0(4) Let p,(t) = pr(Z, = k), and abbreviate it to p,. The forward equations are Po = —APot HP Pe = AP, 1-H ADT Heer (k= 1,2...) Multiply the kth equation by s* and sum over k = 0,1,2,.. I1,{s) to IT; then isn yut—p.)—anr+" ap.) = (F *) és o99) If p,(t) > m, as t + 0, then TT + Xo oH = 0, Therefore ot +3 abbr ns, and {n,} is given by setting»E(i)? ™ = (1—p)p* (k = 0,1,2,...) a1 —; = E(service time)/E(arrival interval) _ is the traffic intensity. Then x, is a probability only if p < 1. For p <1 the sequence {r,} is the geometric distribution %(p). a - Alternatively, equation 6.7 gives this result with 2, = 2, 4, = 1. This queue is in fact a random walk in continuous time with steps _ +1, -1, having probabilities 4/(A+y), /(A+1) respectively, and steps __ taking place as a Poisson process of rate 1+, There is an impenetrable barrier at the origin at which the particle stays for a time which is an _ 6(2) rv and then jumps to +1. BP EXERCISES 1 Accident proneness. Suppose that if no previous accidents have occurred If < o, then the model represents the case in which the lesson has been learnt, but if 6 > a then we have the case in which the shock leads to a _ state in which more blunders are likely. Find the Kolmogorov forward differential equations for p,(t) = pr(N, = k), where N, is the number of accidents in (0, t), and show that I7,(s), the pgf for N,, satisfies pus = (B—a)(1—s)e™™ "Find EN, from this equation (by the method of Problem 6.4). Note that this model reduces to the Poisson process when « = f. _ 2. Suppose that the probability of an accident in (t,t+4) given that /k accidents have already occurred is (a+kd)A+o(4) (k = 0,1,2,...) Find the Kolmogorov forward differential equations for p(t) in this case. Verify that they are the same as for a linear birth process with immigra- _ tion, the birth rate per individual being and the immigration rate being o. _ Solve the equations to show that N, has a negative binomial distribution. 3 Race relations. For the linear birth process with immigration of _ Exercise 2, show that if there are n individuals present at time zero, then _ the population size at time t has pgf sel!— se — 1) 09 If the individuals present at time zero and their descendants are called — natives, and those who enter after time zero and their descendants are called immigrants, prove that if ni > a the expected number of natives always exceeds the expected number of immigrants, 4. The linear birth and death process. If there is no immigration and if the birth rate per individual is 4 and the death rate perindividual isu(a # p), show that the population size at time 1, Z,, has pef (a—by)/(a—cy)), if initially Zy=1, where a=As—p, b=(s—ly, c =(s-l)4, p= exp{(2—y)t}. Determine EZ,, VZ, and the probability of extinction. 5. Do Exercise 4 and Problem 6.3 for the cases in which 2 = y. Check that the results could have been derived by setting » = A+e in the solu- — tions for A # y, and letting ¢ > 0. 4 6. Find VZ, for the population process of Problem 6.3, using the method outlined following the solution to Problem 6.4. community contains i (i = 0, 1, 2, ...,) members at time ¢, then, during a short interval (t, t+ 4),a new member joins with probability(n—i)Ad+0(4); each member independently leaves with probability 44+0(4), and there is no change in membership with probability 1—ni.A+o(4). Set out the Kolmogorov forward differential equations for the distribution of the size, Z,, of the community at time t. If initially the community had no members, show that Z, is Bin © (n,3(1-e" 4), 8. The incoming traffic to a car park with n spaces is a Poisson process of rate A, but only while spaces remain unfilled. In any short time interval of length 4 each vehicle already in the car park independently leaves with probability v4+0(4). Write down the Kolmogorov forward _ differential equations for the distribution of the number, Z,, of spaces filled at time t. ¥ Find the equilibrium distribution of Z,. If the car park is large enough that there is always room for incoming traffic, and if initially k spaces are filled, find EZ,. 9. A single-server queue with constant service times. Customers arrive in a Poisson process of rate 4, and the service time is unity. Let rv Z, be the number of waiting customers, including the one being served, at time 1. Show that if ¥, customers arrive in time interval (t,¢+1) theneee eae . "7 fete Gabe.) tet = Vy (Z,=0) recurrence relations for p,(t) = pr(Z, = k), and show that if G,(s) $6, 19) = {(8—I)polt) + G,(s)} exp{a(s— 1} ee Problem 4.12 and Exercises 3 and 4 of Chapter 4.) sae at Non-Markov Processes in Continuous Time Discrete State Spaces We now consider population processes for which the lifetimes of. members — are not necessarily exponential rvs. The remaining lifetime of an individual _ will depend on how old he is now, and so the future size of the population — will depend not only on present size, but also on the present ages of its members. The process is still Markovian if we include this information, but we would not wish to keep records on every individual throughou their lives. There is no change i in the population size except when an event occurs: when a child is born, or someone dies, or an immigrar arrives or an emigrant departs. We consider, therefore, decompositi based on the times of the first or last of these events. 7.1 Renewal theory We consider first the special case of a renewa process. The function of a single component—for example, a continuously burning light bulb—is observed and it is replaced by another immediately it fails. The components will have lifetimes which are independent {X,}, each distributed like a positive rv X. Then the rv Z,, the time to the 1 failure of the nth component if the first is fitted at time 0, is given by th random walk Z,=X,+...4X,, Zy =0 See Figure 7.1. We take a fixed time r and are interested in the following — tvs: i the number of renewals in (0, a. N, = max{n:Z, < t} the forward recurrence time, 7” = Zy,,,—t and the backward recurrence time, T, =t-Zy, ————SS Zu + 7 Figure 7.1 Now, for a fixed t, Ty Ape By Eg Ng is not distributed as X, since in any realisation of the process the fixed 7 ‘ ie¢ likely to be in a long interval non-Markov since T;* depends on T;. For example, if a light ee Sree Se ee ree “than a week or a good one which lasts about a year—then, if a bulb has survived 6 months, it is of proven quality with about another 6 months’ lifetime. "There is an important and fundamental relationship between the counting process {N,:t€(0,0o)} and the random walk process {Z,: = 0,1,2,...}. The events ‘the number of renewals in time t is at least n’ and ‘the time to the nth renewal does not exceed ¢’ are the same, and so have the same probability. Therefore pr(N, > n) = pr(Z, < 1) (7.1) “Suppose that EX =u, VX = 0%. We can approximate the distribution of __N, for large t by working directly with Z,,; that is, 4 pr(N,t, then N, = 0; if U =u <1, then N, = 1+N;_,, where aN. the number of further renewals in the remaining t—u up to ¢, and so ‘is distributed like N,_,. We are using the fact that the Markov property holds at renewal points, since then it is as though the process were starting Es" = ByEgyyaus = [2° fl) du Eyyyaus™ = [f epdu Este ++ | fudu Bs? II{s) = sf lM, (9) du | flu) du Problem 7.2 Determine integral equations for the renewal function H, = EN,, and the renewal density h, = (d/dt) H,. Solution. We differentiate equation 7.2 with respect to s, and let s > 1. as) _ an, 35 ly SMM dues | fa) Se Therefore H,= {flu dut J, S00H,. du We differentiate this with respect to t: h, = fO+SOH.+ f° flwh,_, du =S0+ [ Oh, du since Hy = EN, = E0 = 0. Since N, ~ t/u ast > 20,H, ~ t/wandh, ~ 1/p. Problem7.3 Show that h, is the instantaneous transition rate defined by pr{a renewal occurs in (t,t+4)|a renewal occurred at 0} = h,A+o(A) Solution. If we write equation 7.3 in the form h,A+0(4) = {f04-+0(4)} + f (Fw) du} {h,,+0(4)} then the right side is pr{the first renewal is in (t, t+ A) or it is in (u,u-+du) for some ue (0,1) and there is a renewal in an interval of length A aftera further t—u} ie. it is the probability that there is a renewal in (t,t + 4) given thet was a renewal at 0. ; Problem 7.4 When the lifetime rv is 6(2), that is, ex; ial wit i ponential with p mee 2, show that rv N, is A(22), that is, Poisson with mean At, and 1 is 2. Solution. We use the definition of h, given by Problem 7.3. If h, then by Problem 5.2 N, is Azz), and em i eee , is Az), and by Problem 56 the intervals The other way round: we put Sf) =te-*™ O 0), or the last renewal before t is in (v,v+do)(0 < v < 1) for some v €(0,t), and the next renewal is in (t+x,t+x-+ A) ie. in (t+x—v, t+x—v+4) later} G9. (¥)4 +04) = {f(t+x)4+0(4)} + fhe do {f(t+x—0)4 +0(4)}. fe set u = t—v in the integral, divide by A, and let A + 0. Then 3 a (8) = flt+x)+ J" hy, flux) du (74) _ We must suppose that f(x) + 0 as x + oo. Equation 7.4 is true for fixed t. t + 00, then h,_, + u~1, so (3) > as) = [° flux) du ts A, = &, since Ay = Iq(s) = Es®° = 1 =a =s) 5 JF saw = j{I-Fe) oo _ Problem 7.6 Find the distribution of the backward recurrence time T,, and its limit as t + co. - pr{T, €(%,x+4)} ‘there is no renewal in (0,1), in which case T> = t, or there renewal in (t—x,t—x +4) (0 < x < t)and there areno more renewals in (t—x, t)} Nowpr{norenewalin(u,v)|arenewalatu} = pr(X > v—u) = 1—F(o—w). Therefore p(T, = t) = pr(X >t) = 1-F() 9, &) =h_,{1-F(X)} O 00,F(t) > landh,_, > p-}, therefore the distribution of T, tends to a density g(x) = {1—F(x)}/u (0 < x < 00), the same lim istribution as that of T,". od 7.2 Population processes If instead of replacing a failed component by — another we replace it with X components, where X is an rv having pgf G(8), then the situation is that of the family tree described in Problem 4.14, except that the lifetimes of individuals are independent rvs each having density f(x) (0 < x < 0). Problem 7.7 The continuous-time branching process. Suppose that at _ time 0 the process starts with the birth of a single individual, the founder. _ Find an integral equation for the pf, IZ(s), of the size, Z,, at time f, of the population. a Solution. We consider a decomposition based on the time, U, of the founder's death. Let N be the number of offspring left by the founder. Then Ey s¥ = G(s). If U = u, then, for t u z, Be ZO +... Z0, (N = 1,2,...) a ei (N =0) where each ZY, is independently distributed like Z,_,. Therefore, for t > u, by Problem 2.14, Ezju-veiS” = G{M,_(9)} Therefore Ts) = Ez,s* = EyEziyaus* = [° flu) du Ezyyays™ = J) £00) du G1, _(0)} du-+[* fu) du Bs! = s(1-FO}+ ff FONGET, 9} au O05) Problem 7,8 Determine ITs) in the special case in which the number of offspring is always 2 and the lifetime rv is &(4).Solutio ME i ah eae tl a al es feGh waa Je (0 < u < 00). This is the linear th process, a special case of Problem 6.3. Equation 7.5 becomes wath Ts) = se-**+ |" de“ ™{T1,_(9)}? du Multiply by e*, set v = t—u in the integral, differentiate with respect to t, divide out e*, and abbreviate JT,(s) to 17. Then ; al re Gy 7 M-1), a iol |a7-1| Adt= aay = (ea-z)a = dlog—- es |m-1| _ 4 log’, — = 2t-+constant = sew a low TI,(s) = s, so I,(s) = isa-e ie. Z,—1 is Y(e~*), a geometrically distributed rv. o een the arrivals of customers are independently and identically _ distributed continuous rvs, that the service times of customers are inde- ‘tly and identically distributed rvs, and that there is just a single . Label the kth customer C,. Denote by ¥, the time between the s of C, and C, , ,; by X, the service time of C,; and by W, the waiting (which excludes the time in which heis being served) of C,. Ifthe density f the rv V, = X,—Y, is g(v) (—co < v < oo), and the distribution func- _ tion of W, is F,(w), show that v : Faw) = |" For—v)o(o) do Solution. Assume that C, arrives at t = 0 and finds no one ahead of ‘him; therefore W, = 0. Customer C, is in the queue for a time W+X,. “If %> +X, then W,,, = 0; that is, if Cys; arrives late enough he has no waiting. If ¥, < W,+X,,thenW,, = W+X,-Y = With. is, W., = max(W,+ V,,0). Now Fi(w) = 0(w <0, for all k), There. , for w > 0, fi rarer Pasa is Fest) = pre Sw) = = prmax+ ¥,0)< w) = hth ~ jh pr{h, €(v,0+do)}pr(W, < w—v| V, = 0) = J" a) dv F(w—v) since pr(W, < w—v|K, = v) = 0(v > w) and using the independence of _ W, and Vj, ae ‘As k — 00, Fi(w) + a limit F(w) which satisfies Fw) = ie F(w—v)g(v) dv. It can be shown that F(w) = po S ¥, < w for all n) (w > 0); and if EX, > O then F(w) = 0, i.e. the queue grows without limit; but if BY; < a then F(w) + 1 as w — oo so the limit F(w) is a df. Problem 7.10 The busy period of a single-server queue with Poi arrivals. Suppose that customers join the queue at times which form a_ Poisson process of rate a, and that the service times are independent rvs _ having characteristic function (8). Find equations for the pf G(s) re the number of customers served during a busy period, and for the of (0) for the busy period, B. ; Solution. The starting times of the busy periods—that is, the time points when a customer arrives and can be served without waiting—are ren points. The ends of the busy periods—that is, the time points when server becomes idle—are also. The busy periods are therefore indepen- dently and identically distributed rvs. The number, A, of customers who arrive during the service time, Bi of the first customer, C,, is P(aX), so os — E,jx=x5* = exp{—ax(1—s)} y These customers are C>, C3, .... C,,,. Without losing generality y imposea last-come first-: served queue discipline. ‘Then C,,,,, isserved second. — While C,, , is being served, other customers arrive and will all be served before C,. The number served starting with C,,, before C, is reached is a rv N,, distributed exactly like N, the number served in a busy peri Similarly, the number, N,_,, served, starting with C,, before C,_, reached is also distributed as N, and is independent of N,. Therefore, — given A, N =14N,+N,+...4N, where the 1 is for C,’s service, so Enanas” = s{G(s)}*7 Se? 4 f : orbs population EE af tino'e ¥ inltally there steno Bemhenon G(s) = Eys" = Bx, ay8" = SExEgr=x{G(9)}* POLS an ’ = sEyexp[ —aX{1—G(s)}] = s¥[ia{1—G(s)}] Gs) = eet ee TT,_,{3)®,_,(s)ae~™ du If N =n, then B = X,+...+X,, 80, by equation 2.28, where the immigration rate is «, and I7(s) is the pgf for the process with o : no immigration from a single member given in Exercise 4 of Chapter 6. : H(0) = G{V(O)} = WON/Lia{ 1 —6(0)}] Qo Solve this equation for (s) and verify that the solution is that derived in This problem can be formulated as a random walk and, as such, a special Problem 6.3. case was solved in Problems 5.7 and 5.8, Show that as t + 00, if yu. Derive, by a decomposition based on the admission of the _ events of a Poisson process having parameter 2, show by a decomposition first new member, the pgf for the membership size at time ¢, conditional - based on the final A, that p,, the probability that the counter is locked at ‘on the founder still being alive then. Hence find the pgf for the member-_ time ¢ satisfies ship size just after the founder’s death. Fi+(+Ap, = 2 7. For the model of accident proneness described in Exercise 1 of : 4 Chapter 6, by means of a decomposition based on the time of the first agen ne potution, accident, show that the pgf for the number of accidents in (0, t) is "3. Find the solution, 17(s), of equation 7.5 in the special case in which ee t the number of offspring of an individual is equally likely to be 0 or 2, fase“#-*—(B—a)(1 sje} /{Bs—(B—a)}. __ and the lifetime distribution is &(2x). What is the chance that the population Note that this reduces to the pgf for a Poisson rv when « = f. ~ ig extinct by time ¢, if it starts at time 0 with the birth ofa single individual? 4, The linear birth and death process with immigration. Show by a decom- _ position based on the arrival of the first immigrant that the pgf, #(s), 84t/At einen steps. Tae from ee 3. a 2, q0-aax = wa e are Markov processes {Z;:t ¢ T} having continuous time para- e meter spaces and continuous state spaces, and for which a small change in Z (ay? " t results in only a small change in Z,. A realisation can be thought of as VZ, = yr fPatx? = = Apa the path ofa particle moving very erratically in a continuous medium, v depending only on its current position. If we let Ax and At —0 in such a way that (Ax)/At > « and ae important class of diffusion processes are the Gaussian processes. fig Veg If {Z,:t€T} is a process whose state space i the real line, R, and if its p= 3+ 34 9=5-7,4* meter space, T, is any subset of R, then it is a Gaussian process if, for a finite n, 2, am Z,,) has a multivariate normal distribution. A where a and f are constants, then EZ, + ft and VZ, ~ at. By the c wussian process is stationary if its covariance function g(s,t) = g(t—s). limit theorem (Z, — Bt)/,/(at) > an N(Q, 1) rv. Wiener (or Brownian motion) process with parameter 4, {Z,:t € R}, is the Gaussian process with independent increments having EZ, = 0 and Problem 8.2 Show that u, ,, where - g(s.t) = Amin(s,, (2 > 0). | fusion Processes u,, dx = pr{Z, €(x,x+dx)} em 8.1 If {Z,} is Wiener with parameter «, show that {Y,} = ‘Z.2a:} is a stationary Gaussian process, and determine its covariance ion, g(s, 1). lution. If Z, isa normally distributed ry, then so is a(t)Z,(9, Where a and Barscal functions of f, since, for fixed 1, a(t) and b(t) are real constants. Solution, From equation 2.12, by a decomposition based on the last t e process {Y;} is therefore Gaussian, since Z, is normal. Now EY, = e “EZ. =0 since EZ, = 0 for all r. Therefore a(t,t+t) = covlY, ¥,.) = EY, Ka.) we Efe" *Z 6 PZ Sasa} =e 9cov{ ZZ ecient ' = e424 min {624 @2ME+9} eA ae gr coe not depend on t. oO mi Merton = ut (At 5 + O(a? ‘ rene procesecs can arise as the limit of a random walk. Let us consider eu, in the unrestricted random walk, but suppose that the particle takes : Uy any = M+ (— Ax) aa y—axp 4 o(axy ependent short steps of length 4x, to the right with probability p, to the left with probability q = 1—p, after short time intervals of ou gth At, It is a delicate matter how we let both 4x and dt tend to zero. Hany = HC) Se Ho) 5+ OC satisfies the forward partial differential equation Ou a eu ax a ~ Pex pr(Z,,.4, = x) = pr(Z, = x—Ax)pr(Z,, 4, = x|Z, = x— Ax) +pr(Z, = x+Ax)pr(Z,.4, = x|Z, = x+Ax) | Therefore Maar ar = Menace Pt Met aeed We expand each term in a Taylor series up to O(Az)* or O(Ax)?, abbreviate u,,, to u.wr (A) + O18)? = Mp +a)+(—p+ qa) ++ gids Fe ota) P+q=1,-pt+q= oe therefore ou — _B f(x?) du | 1 f(Ax)?) du (Ax)? = Be eee +0(ax@2") Ax and At +0 in such a way that (Ax)?/At > a, we obtain tion 8.1. d 7m 8.3 Show that the Brownian motion process satisfies equation -_ Solution. Here EZ, = 0, so f = 0; and VZ, = amin(t,t) = at. There- fore, since the displacement Z, is N(0,a1), from equation 2.29 we have P(x, y) dy = pr{Z,€(y,y+dy)|Z, = x} 1 1 - Jean"? {0-27} dy (-~
oin (Axj/Ar—+y, ade +B! : Solution. Let X¥ = X,—a and k = i-a If X¥ = Xty = kAx = say, then i (k+ Ax = x+Ax with probability ? 2 aka = 1 —x/ad (k-1)Ax = x—Ax with probability a 3+tk/a = {1 +x/aAx) ea Xa = Xie nae = ut (ant a O(At? x 1 u 1 Ltt yuct tac 1 i.e Ou A +5(4bs {us canst +(x) 53 + Ol a (ear (2)2) scar outa Therefore ae a oan) = (a lz So (F3)+0 (ax *) os? Therefore ou é @u a Ba (uta 5 a a This limiting equation is is the diffusion equation for the Ornstein-Uhi Process, which is defined to be the Gaussian process with EZ, = 0 covariance function g(s,t) = ae~**-"! (¢ >0, B > 0). Problem a showed a method of constructing it from a Wiener process. EXERCISES 1. If {X, : te [0, 00)} is the Wiener process with parameter A, for e:{Z,:te[0,0)} find EZ, covZ,,Z)) (< n Bate tietier tion process is stationary in the wide sense, and whether is Gauseian. =at+X, (@>0, (ii) Z,= X,,,-X, (@ >0), =(1-9Xy4-9 O )), Z, = Y,,, —Y,, where Y, is Gaussian with EY,=a+ft, cowY,,¥,,,.)=e7" (y > 0) . Consider the diffusion limit of an unrestricted random walk on the axis. If at time t, X, = x, then at time t+At (n—1)Ax = x-Ax with probability p, atta = nx = x with probability r, (n+1)Ax = x+Ax with probability p, and r, are continuous functions of x, and 2p, +r, = | for every =X, Let u,,dx = pr{X, ¢(x,x+dx)}. If Ax and Ar both tend to zero in a way that (Ax)?/Ar — , a constant, show that in the limit u = ou 2 Gn Pe.) Consider a particle which starts at the origin and carries out a random according to Problem 8.2. If @,(6) is the characteristic function for and if y(0) is the cf for a N(f, a) rv, show by a decomposition of 4, 4 (0) d on the last At that @,(6) = ev _In a model for gas diffusion through a porous membrane there are 2 A and B, cach containing 2c molecules. These molecules are of 2 there are 2c black ones and 2c white ones. After each unit interval are interchanged. The rv X, is the number of black molecules in A time n. Consider the Markov chain {Z,:n = 0,1,2,...}, where = X,—c. Find difference equations in k and n for pf = pr(Z, = k) 1 a decomposition based on the last step. _ Consider the diffusion process arising as follows. If at time instant t, z, then at ¢+At, Z,,4, = z—Az, z or z+Az, where the process has such a way that (Az)?/At > y and (c At)" — f, show that in the limit satisfies the Ornstein-Uhlenbeck or differential equation ou # penny where u,,dz = pr{Z, €(z,2+dz)} 5. Suppose that {Z, :€[0,0)} is the Omnstein-Uhlenbeck process. If Z, given that Zo = z is N{ze~™, (1 —e~?#)/(2)}, show that u,.dx = pr{Z,€(x,x-+dx)|Zo = z} satisfies equation 8.2. Note that the equilibrium distribution of Z, is given by letting t > co, and so is N(0,7/28), ba does not depend on z. Recommendations for Further Reading A fairly elementary introduction to probability theory is oy Wil is of General treatments of stochastic processes at about the level of Problem Solver are given in The Theory of Stochastic Processes David R. Cox and Hilton D. Miller, published by Methuen in 1965, and i The Elements of Stochastic Processes with Applications to the Natu Sciences by Norman T. J. Bailey, published by John Wiley in 1964. Some specific applications are made in Queues by David R. Cox and Walter L. Smith, published by Methuen in 1961, The Theory of Storage by P. A. P. Moran, published by Methuen in 1959, and Stochastic Models for Social Processes by David J. Bartholomew, the second edition of which was published by John Wiley in 1973.bing barrier 23 orbing state 23, 68 Proneness 74, 85 state 51 ‘@, Bernoulli distributed 8 jackward equations 68 ulli distribution, trial 8 Bin, binomially distributed 14 Branching process 48, 60, 81, 84 n motion process 86 ur-parking process 75 Central limit theorem 20 _ cf, characteristic function 16 Chapman-Kolmogorov equation 39, 67 Coin-tossing process 8 unicating states 53 tion for food 75 nal distribution 9 mal probability 9, 12 ition based on the first step 24 m rule 11, 13 asity, probability density function 7 df, distribution function 7 - Diffusion approximation of a random walk First passage time 51 Fokker-Planck equation 88 Football match 2 Forward equations 68 4, geometrically distributed 18 Gambler's ruin 23, Gamma distribution 18 Gas diffusion processes 43, 88, 90 Gaussian process 86 Immigration-emigration process 72 Impenetrable barrier 29, 60 Independence 9, 12 Independent increments 22 Instantaneous state 68 Instantaneous transition rate 79 Insurance models 2, 6 Irreducible MC, class 53 Joint distribution function 9 Kolmogorov differential equations 68 ‘Kolmogorov’s theorem 53 Kronecker delta 39 Law of large numbers 20 Leads to, > 53 Library book loans 4, 37, 57 4, mean of a distribution 7 Machine breakdown 4, 37 Marginal distribution 9 MC, Markov chain 35 Markov process, property 22 Matrix of transition rates 67 ‘Mean 7 Miracle 3 Moments 16 'N, normally distributed 16 NB, negative binomially distributed 18 ‘Newspaper sales 66 Nuclear chain reaction 58 ‘Null-recurrence 53 0 0,~ 67 Oceanography 3 One-step transition matrix 35 Orderly process 62 /Ornstein-Uhlenbeck process 89, 91 ®, Poisson distributed 14 Period, periodicity, periodic state 51 psf, probability generating function 13 Poisson counting process, Poisson process a Population processes 48, 60, 68, 75, 81 Positive recurrent 53 r, probability function 6 Probability density function 7 Probability distribution 6 Probability of success 8 Programmed learning 34 Qmatrix of transition rates 67 ‘Queuing processes 1, 45, 59, 73, 75, 82 Race relations 74 ry, random variable 6 Random walk 19 Realisation 2 Recurrence time 77 Reourrent state 51, 52 Reflecting barrier 28 Renewal density, function 78 Strong law of large numbers 20 Teeth 5 Thinned process 39 Time independence 25, 27, 35 Time to absorption 27 Traffic 5 Traffic intensity 46, 74 Transient state 51, 52 Transition matrix, probability 35 Transition rate 67 Trial 8 Unrestricted random walk 19, 52, 54, 63, 64 4 V, variance operator 7 Variance 7 Wiener process 86 Yule process 7210. 11. ‘zs 13. 14, 15. 16. Calculus of Variations J. W. Craggs Laplace Transforms J. Williams Statistics 1 . A. K. Shahani & P. K. Nandi Fourier Series and Boun- dary Value Problems W. E. Williams Electromagnetism D. F. Lawden Stochastic Processes R. Coleman Fluid Mechanics J. Williams Groups D. A. R. Wallace ISBN 0 04 519016 X Printed in Great BritainOrdinary Differential Equations J. Heading Calculus of Several Variables L. Marder Vector Algebra L. Marder Analytical Mechanics D. F. Lawden Calculus of One Variable K. E. Hirst Complex Numbers J. Williams Vector Fields L. Marder Matrices and Vector Spaces F. Brickell S$3SS3900ud DILLSVHOOLS